From: redpointai
Snowflake, led by Head of AI Baris Gakkan, focuses on developing infrastructure that supports AI tools in production, offering significant insights into real-world AI use cases for businesses [00:00:15]. Snowflake’s comprehensive AI product portfolio includes Cortex AI, the data cloud, and their own Large Language Model (LLM) Arctic [00:00:07].
Snowflake’s AI Product Portfolio
Snowflake’s core AI offering is Cortex, an inference engine that runs various large language models, including Arctic, Mistral, and Meta models [00:10:29]. Built on top of Cortex are two key products:
- Cortex Search – A high-quality search system essential for Retrieval Augmented Generation (RAG) [00:10:39]. It uses Arctic Embed, Snowflake’s embedding model, which is noted for its efficiency and state-of-the-art performance [00:11:00]. Cortex Search is a hybrid system, combining vector search and lexical keyword search to reduce hallucinations [00:12:14].
- Cortex Analyst – A business intelligence (BI) product designed for interacting with structured data [00:10:49].
Snowflake’s strategic acquisitions, such as Neva (which brought its CEO and Neva technology for embedding models and search) and Trulens (an observability and LLM evaluation platform), have been instrumental in building out their AI capabilities [00:14:06], [00:28:55].
Generative AI for Business Applications
Snowflake’s customers leverage AI for three primary use cases [00:03:21]:
AI for Business Intelligence (BI) and Text-to-SQL
This use case focuses on creating BI-type experiences using natural language to SQL to data [00:10:09]. Snowflake’s Arctic LLM was specifically developed to excel at SQL generation and follow instructions effectively [00:01:19].
Challenges in AI for BI
- “Iceberg Problem”: Demos are easy to build, but real-world data is “incredibly messy” [00:04:30].
- Data Complexity: Companies often have tens of thousands of tables and hundreds of thousands of columns with inconsistent naming conventions (e.g., “rev one,” “rev two”) [00:04:47].
- Accuracy and Hallucinations: LLMs might hallucinate column names, get complex joins wrong, or generate non-executable queries [00:07:03]. LLMs will provide answers even when they shouldn’t [00:07:40].
- Trust in Business Criticality: For BI uses like reporting revenue to a CFO, 90-95% accuracy is often insufficient; complete trust is required [00:05:40].
Solutions and Approaches
- Stitching Multiple Systems: Cortex Analyst combines three to four LLMs working together to answer BI questions [00:05:05].
- Self-Healing Systems: These systems generate SQL, check its validity, and learn when to ask for clarifications or abstain from answering questions it cannot answer [00:05:25].
- Human-in-the-Loop: Systems are being built to ensure human oversight and correction [00:05:51].
- Verified Queries: Customers can create “verified queries” which, if used, are returned with increased confidence via the API [00:06:14].
- Prioritizing Precision over Recall: Snowflake consciously makes a product choice to prioritize precision, meaning they focus on providing correct answers even if it means not answering all questions [00:07:55]. This contrasts with an open-ended approach suitable for analysts, but not business users [00:08:21].
Data Extraction and Text Analytics
This involves using natural language to extract data from text for later processing [00:03:48]. This is seen as a well-working application of AI [00:08:48].
Use Cases
- Analyzing Unstructured Data: Processing documents, PDFs, sales call logs, customer support tickets, and other semi-structured text in batch [00:03:35].
- Employee Survey Analysis: An LLM can summarize open-ended employee survey questions, categorize responses, and provide example quotes, tasks that previously took hours [00:09:09].
- Simplifying Workflows: Snowflake focuses on providing simple, easy-to-use, task-specific functions that reduce the need for customers to manage prompts and data pipelines [00:09:51].
Other Enterprise AI Applications
- Chatbots: Building chatbots for interacting with unstructured data [00:03:32].
- Internal Productivity Tools: Internal use cases for AI, such as summarizing sales conversations, employee assistants for internal documents, and documentation-oriented chatbots, are common and carry lower risk from hallucinations [00:12:23], [00:30:20].
- Enterprise Search: Upgrading existing enterprise search stacks with AI capabilities [00:12:35].
Challenges in Enterprise AI Deployment and Solutions
Data Governance, Security, and Access Control
A major initial step for large companies is ensuring comfort with the chosen AI platform, focusing on trust, data security, respect for data governance, and acceptable policies [00:13:10].
- Running AI next to data: Snowflake’s advantage lies in running all models inside Snowflake, right next to customer data, ensuring data security and adherence to granular access controls [00:16:37].
- Granular Access Controls: Snowflake’s existing granular access controls for database objects extend to AI systems. For instance, search systems filter results based on user access to documents [00:17:03]. This prevents data leakage and ensures role-based information access (e.g., in an HR chatbot, responses differ based on who asks “who got promoted?“) [00:17:21].
Model Selection and Optimization
- Starting Point: Enterprises are advised to begin with large models and RAG solutions for proofs of concept (POCs) [00:19:23].
- Optimization for Production: For production systems, fine-tuning smaller models offers latency and cost advantages [00:19:36].
- Custom Models: Building custom pre-trained models makes sense for companies with large amounts of unique data, especially in regulated industries like healthcare, where full control over training data and understanding specialized language is crucial [00:19:51], [00:20:22].
- Brand vs. Capability: A “somewhat surprising” pattern is that customers often select models based on brand recognition rather than actual capability [00:21:36].
- Cost: While LLM costs are a concern, they haven’t been a major blocker for enterprise adoption yet, as most current use cases are internal, leading to non-massive volumes. Costs are also rapidly decreasing [00:25:20].
Evaluation and Observability
- Transition from POCs to Production: Enterprises are slowly moving from POCs to production for internal productivity and analysis use cases, but external use cases still require more confidence [00:15:07].
- LLM Evaluation: Evaluation of LLMs is still not mature [00:28:11]. Trulens, an acquired open-source observability and LLM evaluation platform, is critical for customers to evaluate systems at scale, often using LLMs as judges [00:14:18].
- Guardrails: Companies like Snowflake are building additional guardrails into models (e.g., Cortex Guard using Llama Guard) to manage unscripted responses and ensure brand and policy alignment in uncharted territory [00:27:33].
Future Outlook
- Agents: Agentic systems are an “uncharted new territory” with “super exciting” potential [00:28:24]. While early, customers are beginning to implement them for specific use cases like BI workflows, where a system orchestrates multiple steps (SQL generation, evaluation, data retrieval, reasoning) [00:39:17].
- Specialized Agents: The vision includes “many small, specific agents” set by natural language to perform tasks on behalf of users, like checking websites regularly for unique needs [00:38:29].
- Ecosystem Play: Snowflake aims to enable a full ecosystem for customers by making it easy to use and share AI-ready data within the data platform [00:35:12]. This allows companies like Morgan Stanley to build chatbots that leverage existing data from the financial ecosystem (e.g., S&P, FactSet) with built-in governance and security [00:35:07].
- Application Space Maturing: The market is transitioning from providing building blocks to demanding end-to-end products, with customer service being an obvious application area [00:45:30].
- Startups in AI Infrastructure: Despite the convergence towards end-to-end platforms, there are still “massive” opportunities for startups across various aspects of the AI infrastructure stack, driven by ongoing innovation [00:33:33]. The inference stack, in particular, continues to develop rapidly, translating directly to cost reductions [00:35:52].
- Assistant Platforms: The assistant space is challenging for startups because it needs to be a platform but cannot start as one. The key for startups is to “nail a couple of really deep use cases” rather than attempting many small ones [00:43:41]. Examples include specialized “engineer” or “salesperson” agents [00:44:06].