Tubegraph

Mistral AIs open source LLM development

6 min read

From: redpointai
Mistral AI, co-founded by Arthur Mensch, is at the forefront of the AI landscape, particularly known for its contributions to open-source large language models (LLMs) [00:00:10]. The company aims to provide developers with the most relevant platform for their AI needs [00:05:29].

The Name: Mistral

The name “Mistral” has two stories [00:02:02]:

  • Official Story: It relates to the French pronunciation of Artificial Intelligence (“Intelligence Artificielle”), where “Mistral AI” contains “AI” (as in “I A”) [00:01:08].
  • True Story: The team struggled to find a name, considered forest names, and eventually settled on Mistral because it sounded good. It refers to a specific cold wind blowing in the south of France [00:01:22]. Arthur Mensch describes it as a “Wind of Change” [00:01:41].

Open-Source Philosophy and Product Strategy

Mistral strongly believes that AI is an infrastructure technology that should be modifiable and owned by customers, with open source models prevailing in the long term [00:03:36]. They aim to establish a business model that sustains open-source development [00:03:50].

Their initial model distribution via torrent was a deliberate nod to how LLaMA was initially shared, which resonated well with developers [00:01:54].

Mistral operates with a dual offering strategy:

  • Open Source: They provide models like Mistral 7B, which is noted for its efficiency and remains a leading model in that regard [00:04:35]. These are typically models that are just behind their very best commercial offerings [00:05:10].
  • Commercial (Closed Source): This includes models like Mistral Large, Small, and Embed [00:04:24]. Mistral Large is offered as a portable solution, providing customers access to the model weights, which is similar to the usability of an open-source model [00:05:03]. This approach allows enterprises to deploy models where their data resides, addressing data governance concerns and enabling specialization and custom applications [00:06:00].

Enterprise Strategy and Partnerships

Mistral’s core strengths lie in training and specializing models [00:07:00] [00:07:05]. While they are building their own inference pipeline, they also leverage partnerships for distribution and adoption [00:07:27].

Notable partnerships include:

This multi-platform strategy is driven by the need to meet enterprises where they operate and facilitate adoption [00:08:10]. Smaller, digital-native companies often engage directly with Mistral, while larger European enterprises prefer to use existing cloud credits via partners like Azure to simplify procurement [00:09:36].

Future of LLMs and Efficiency

Mistral continues to push the efficiency frontier of LLMs [00:10:25]. Key areas of development include:

  • Model Compression: Mistral 7B demonstrated significant compression, and more improvements are expected [00:10:29].
  • Controllability: Significant research is still needed to make models more controllable and follow instructions precisely [00:10:44].
  • Architectural Improvements: While Transformers are dominant, Mistral is researching more efficient architectures (e.g., sparse attention) [00:11:08]. The co-adaptation of the ecosystem (training algorithms, hardware, debugging) to Transformers makes non-incremental architectural changes challenging [00:14:50].
  • Deployment & Latency: Goals include deploying models on smaller devices and improving inference speed, which will unlock new applications involving LLMs as basic “bricks” for planning and exploration [00:11:27].

Despite having fewer GPUs (1.5K H100s) compared to large players like Meta (600K GPUs), Mistral focuses on efficiency and a high GPU concentration per person to stay competitive [00:12:01]. Their past work, like Chinchilla (showing significant model size reduction for same performance) and Mistral 7B (achieving a factor of six improvement), underscores their ability to innovate with less compute [00:13:56].

AI Regulation and Policy

Mistral advocates for regulating AI safety from a product safety perspective, similar to how general software safety is handled [00:17:17]. They believe the EU AI Act, while manageable, misses the core problem by focusing on technology-level regulation (e.g., flop thresholds, mandatory evaluations) rather than application safety [00:17:50].

According to Mensch, LLMs are like programming languages; their safety depends on how they are used in a product [00:18:07]. The challenge is in evaluating stochastic models and rethinking continuous integration/verification for AI products [00:19:39]. Mistral sees this as a technological and product problem, where companies should provide developers with tools to ensure application safety [00:19:47].

They suggest policymakers should pressure application makers to verify their tasks are solved correctly (e.g., safety testing like cars) [00:21:17]. This would create a “second-order pressure” on foundational model makers to provide controllable models [00:21:34].

Mistral also acknowledges discussions around transparency of training datasets, with the caveat of needing to protect trade secrets in a competitive landscape [00:18:45].

Global vs. Local Models

Mistral’s approach to national sovereignty in AI is through portability of their technology and being a multilingual company [00:23:25]. They aim for models to be great in every language, starting with French, where their models are currently among the best [00:23:46].

The ability to control and modify the technology, as offered by Mistral’s platform play (shipping models), should be sufficient for countries to feel confident in their control over the technology, as opposed to relying solely on Software-as-a-Service (SaaS) offerings that could pose a sovereignty problem [00:25:13].

The Genesis of Mistral AI

The confidence to start Mistral AI came from several factors [00:26:01]:

  • Prior experience in the field [00:26:10].
  • Securing a strong initial team [00:26:14].
  • Identifying a significant talent pool in Paris [00:26:30].
  • The increased awareness in the VC world due to ChatGPT [00:26:34].
  • Confidence in their ability to ship high-quality models quickly [00:26:42].

Application Layer and Data Strategy

Mistral also develops applications on top of its models, such as lchat (internal name) and Entreprise [00:27:05] [00:27:32]. These serve as entry points for enterprises, demonstrating value and helping solidify APIs (e.g., moderation tools) [00:27:12]. This strategy aims to get enterprises started with generative AI, even before they fully understand its core business implications [00:28:06].

Regarding data for fine-tuning, Mistral advises enterprises with large datasets to first use Retrieval Augmented Generation (RAG) [00:29:13]. Fine-tuning data should ideally be “demonstration data” or traces of user interactions to imitate behavior [00:29:31]. This type of data is often not readily available to enterprises, creating an “even field” where companies must rethink their data strategy for copilot and assistant deployments [00:29:58].

Quick Takes

  • Overhyped: Synthetic data [00:30:27]
  • Underhyped: Optimization techniques [00:30:37]
  • Biggest Surprise (Positive): Gaining attention more quickly than expected [00:31:16].
  • Biggest Surprise (Negative): Challenges in hiring for a US office and attracting top talent [00:31:01].
  • Thoughts on Grok Model: Too big; performance doesn’t match the parameter count [00:31:33].
  • Exciting AI Startup (non-Mistral): Dust, focusing on Knowledge Management with a sleek UI [00:32:01].
  • Alternative AI Application: Accelerating Material Science, such as the synthesis of ammonium, which currently lacks a foundational model [00:32:36].

Mistral continues to solidify its documentation and guides to simplify API usage, serving as a primary resource for learning more about their work [00:33:29].

Graph View

Backlinks