Building LinkedIns GenAI Platform

From: aidotengineer

This article outlines LinkedIn’s journey in building and scaling their GenAI platform, focusing on its evolution, critical components, value proposition, and the approach to building AI teams for such an initiative.

Introduction to the Journey [00:00:31]

The presentation details LinkedIn’s experience in constructing its GenAI platform, covering the reasons for its development, the methodology used, and the current state of its construction [00:00:35]. It also explores why such a platform is crucial in today’s agent-driven world and offers tips on how to build and hire for a GenAI team [00:00:45].

Before diving into the platform, understanding the GenAI product experience it supports is essential [00:01:14].

LinkedIn’s GenAI Product Evolution

First Generation: Collaborative Articles (2023) [00:02:42]

In early 2023, LinkedIn launched its first formal GenAI feature: Collaborative Articles [00:01:29].

• Functionality: A straightforward prompt-in/string-out application leveraging the GPT-4 model to create long-form articles, inviting members to comment [00:01:34].
• Initial Components: The team built key backend components, including a gateway for centralized model access and Python notebooks for prompt engineering [00:02:00].
• Tech Stack: At this stage, two different tech stacks were used: Java for the online phase and Python for the backend [00:02:13]. This was not yet considered a platform [00:02:24].
• Limitations: This simple approach lacked the capability to inject LinkedIn’s rich data into the product experience [00:02:30].

Second Generation: Co-pilot/Coach (Mid-2023) [00:02:42]

By mid-2023, development began on the second generation of GenAI products, internally referred to as “co-pilot” or “coach” [00:02:42].

• Example: A popular experience involved looking at a user’s profile and job description, then using a RAG (Retrieval Augmented Generation) process to provide personalized recommendations on job fit [00:02:51].
• Platform Capabilities:
  • Python SDK: A Python SDK was built on top of the LangChain framework to orchestrate LLM calls and integrate with LinkedIn’s large-scale infrastructure [00:03:13]. This allowed developers to easily assemble applications [00:03:35].
  • Unified Tech Stack: The company began unifying its tech stack, realizing the high cost and error potential of transferring Python prompts to Java [00:03:38].
  • Prompt Management: Investment began in prompt management, creating a “prompt source of truth” submodule for versioning prompts and providing structure for meta-prompts [00:03:51].
  • Conversational Memory: Critical infrastructure was built to track LLM interactions and retrieval content, then inject that content into the final product to enable conversational bots [00:04:08].

Multi-Agent System: LinkedIn H Assistant (Last Year/Current) [00:04:31]

More recently, LinkedIn launched its first real multi-agent system, the “LinkedIn H Assistant” [00:04:31].

• Functionality: This system assists recruiters by automating tedious tasks like posting jobs, evaluating candidates, and outreach [00:04:42].
• Platform Evolution: The platform evolved into an agent platform [00:05:08]:
  • Distributed Agent Orchestration Layer: The Python SDK was extended to support a large-scale, distributed agent orchestration layer, handling distributed execution, retry logic, and traffic shifting [00:05:11].
  • Skill Registry: An investment was made in a skill registry, providing tools for developers to publish APIs as “skills” into a centralized registry. This registry addresses skill discovery and invocation problems, making API calls easy for applications [00:05:36].
  • Experiential Memory: Beyond conversational memory, the platform extended to “experiential memory,” a storage solution that extracts, analyzes, and infers tacit knowledge from agent-user interactions [00:06:14]. This memory is organized into working, long-term, and collective layers to help agents understand surrounding context [00:06:35].
  • Operability: Recognizing the autonomous nature of agents (deciding which APIs or LLMs to call), operability became critical [00:06:50]. An in-house solution was built on OpenTelemetry to track low-level telemetry data, allowing for agent call replay and an analytics layer to guide future optimizations [00:07:12].

LinkedIn’s GenAI Platform Architecture

The platform’s components are categorized into four layers [00:07:42]:

1. Orchestration: Manages the flow and interaction of various components.
2. Prompt Engineering: Tools for designing, managing, and optimizing prompts.
3. Tools and Skills Invocation: Enables agents to call external APIs and tools (skills).
4. Content and Memory Management: Handles conversational and experiential memory.

Beyond these core layers, sister teams contribute to the broader LinkedIn GenAI ecosystem [00:07:56]:

• Modeling Layer: Fine-tunes open-source models [00:08:02].
• Responsible AI Layer: Ensures agents adhere to company policies and standards [00:08:06].
• AI Platform/Machine Learning Infrastructure Team: Hosts the models [00:08:12].

Value Proposition of the GenAI Platform

The key value of this GenAI platform is to serve as a unified interface for a complex ecosystem [00:08:23].

• Simplified Development: Developers don’t need to understand every individual component, but can leverage the platform to quickly access the entire ecosystem [00:08:32]. For example, switching between OpenAI and on-premise models can be done by changing a single parameter in one line of code within the SDK [00:08:50]. This significantly reduces infrastructure integration complexity [00:09:02].
• Best Practices and Governance: As a centralized platform, it enforces best practices and governance, ensuring applications are built efficiently and responsibly [00:09:12].

Why a Dedicated GenAI Platform is Critical

LinkedIn believes a dedicated GenAI platform is essential due to the fundamental differences between GenAI and traditional AI systems [00:09:55].

• Blurring of Phases: In traditional AI, there’s a clear distinction between model optimization and model serving, allowing AI engineers and product engineers to operate in separate tech stacks [00:10:04]. In GenAI systems, this line disappears; everyone becomes an engineer who can optimize overall system performance, creating new challenges for tooling and best practices [00:10:24].
• Compound AI Systems: GenAI or agent systems are “compound AI systems,” which tackle tasks using multiple interacting components, including models, retrievers, and external tools [00:10:49].
• Bridging Skill Gaps: The GenAI platform is critical for success because it bridges the skill gaps between AI engineers and product engineers [00:11:10].

Building the GenAI Team

When building a GenAI team, finding ideal candidates is challenging [00:12:25].

• Ideal Candidate Profile: A strong software engineer capable of building infrastructure integration, with good developer PM skills for interface design, and ideally an AI and data science background to understand the latest techniques. They should be quick learners and hands-on [00:11:55]. Such candidates are rare “unicorns” [00:12:30].

Realistic Hiring Principles [00:12:35]

LinkedIn follows these principles:

1. Prioritize Software Engineering Skills: Strong software engineering skills are prioritized over AI expertise [00:12:47].
2. Hire for Potential: Given the fast-evolving nature of the field, hiring for potential rather than just experience or degrees is crucial, as much experience can quickly become outdated [00:13:03].
3. Build a Diversified Team: Instead of seeking a single individual with all qualifications, LinkedIn hires a diversified team (full-stack software engineers, data scientists, AI engineers, data engineers, recent graduates from research universities, and startup backgrounds) [00:13:15]. This collaborative environment encourages engineers to pick up new skills and grow into ideal candidates [00:13:50].
4. Emphasize Critical Thinking: Teams are encouraged to constantly evaluate the latest open-source packages, engage with vendors, and proactively deprecate solutions, acknowledging that current builds may be outdated within a year [00:14:06].

Key Takeaways and Lessons Learned

Tech Stack Choice [00:14:33]

• Python Recommendation: LinkedIn strongly recommends Python. Despite starting with Java and Python, Python was ultimately chosen due to its prevalence in research and open-source communities, and its scalability [00:14:40].

Key Components to Build [00:15:01]

1. Prompt Source of Truth: Essential for robust version control of prompts, critical for operational stability [00:15:03].
2. Memory: A key component for injecting rich data into the agent experience [00:15:26].
3. Skills: In the agent era, uplifting existing APIs into easily callable skills for agents is crucial, requiring surrounding tooling and infrastructure [00:15:42].

Scaling and Adoption [00:16:04]

• Start Small: Instead of building a full-fledged platform from the outset, focus on solving immediate needs. LinkedIn began with a simple Python library for orchestration and gradually grew into the comprehensive platform [00:16:07].
• Focus on Infrastructure and Scalability: Leverage existing enterprise infrastructure, such as LinkedIn’s messaging infrastructure, as a memory layer, which proves both cost-efficient and scalable [00:16:29]. This aligns with using existing enterprise systems for AI integration.
• Prioritize Developer Experience: The platform’s ultimate goal is to enhance developer productivity [00:16:46]. Its adoption is key to success. Design the platform to align with developers’ existing workflows to ease adoption and ensure success [00:16:56]. This contributes to building user experiences with AI.

For more detailed technical information, readers are encouraged to check LinkedIn’s engineering blog posts [00:17:12].