From: redpointai
Notion AI is designed to augment user workflows within the Notion platform, helping with tasks ranging from writing and editing to information retrieval. Understanding how users interact with these AI features has been a continuous learning process for the Notion AI team [00:16:54].
Collaborative Human-AI Interaction
From the outset, Notion observed a core theme in user interaction with AI: humans and AI consistently work together to iterate and achieve better outcomes [00:12:00]. Users often generate a draft with AI and then refine it, or they iterate on their own drafts to fix mistakes and improve clarity [00:04:18]. This led to the addition of conversational follow-ups for generated content, allowing users to ask the AI to “make it shorter” or “more punchy” [00:04:30]. This iterative approach is a key aspect of HumanAI interaction.
Identifying and Integrating New Features
Notion employs several methods to learn about user interaction and discover new use cases:
-
Internal Dogfooding The Notion team heavily uses its own product, including AI features, in a process called “dogfooding” [00:08:05]. This internal usage, particularly during the ideation phase, provides a rich source of information for understanding useful ways to apply the AI [00:17:14]. For example, an “annoying prototype” of the Q&A feature, which responded to every internal question, forced the team to rapidly iterate and improve the quality of outputs because they were “so face-to-face with it every day” [00:09:58]. This constant exposure made it evident whether the feature was useful or not, adding pressure to improve the model quality [00:10:52].
-
Early Testers and Ambassadors Notion also works with early testers, Notion ambassadors, and partners. These external users often take initial features and enhancements in Notion AI, especially those requiring prompt engineering, and apply them to unanticipated use cases [00:18:00]. An example of this was the discovery of translation as a popular use case for AI autofill outside the US, leading Notion to integrate translation as a built-in prompt [00:18:25]. This method helps in using AI for personalized user experiences.
Pre-built vs. Custom Prompts
Notion’s approach to user interaction balances guiding users with providing flexibility:
-
Guided Experience: Notion provides pre-built prompts (e.g., “summarize this page,” “fix spelling and grammar mistakes,” “improve writing”) [00:04:46]. These serve as templates and are the most popular use cases, especially for Notion AI Writer and Autofill [00:19:15]. Summarization, writing improvement, grammar correction, and translation are particularly widely used [00:19:34].
-
Customization for Power Users: Users can also access a “custom prompt” field for more specific requests [00:18:37]. This extensibility allows Notion to discover new common creative process and experimentation with AI use cases, which are then built into pre-baked formulas or templates over time [00:19:08]. Power users often start with pre-built options for inspiration, then hand-write and reuse their own specific prompts to fit their unique workflows, such as a newsletter writer generating social media tags for new issues [00:20:13].
User Education and Discovery
Notion recognizes the “blank canvas problem” where users may struggle to know how to interact with AI [00:40:40]. Pre-built prompts help overcome this by showing possibilities, getting “creative juices flowing” [00:21:09]. Notion considers suggesting different types of revisions based on what the user is working on, or after AI has generated something, to reduce the “mental work” of coming up with a prompt [00:22:12].
Challenges in Understanding User Patterns
Despite extensive internal testing, Notion encounters diverse user patterns and challenges in deploying AI in Notion:
- Varied User Demographics: Notion’s internal usage might not be fully representative of all users, which include individuals, students, and organizations with diverse ways of using the product (e.g., some still use Google Docs in addition to Notion) [00:17:24].
- “Meta” Questions in Q&A: A challenge with Notion Q&A was anticipating users asking “meta questions” about Notion itself, such as “How can I share this page with Jack?” or “How many people are in my workspace?” [00:24:19]. These questions are not answered by content within a user’s workspace [00:24:24].
- Temporal Queries: Another challenge was handling questions involving time ranges, like “What is the marketing team working on this week?” [00:24:47]. Such queries require the AI to understand continuous updates over time rather than just looking at static documents [00:24:57].
These complexities highlight the need for sophisticated evaluation methods and internal tools to understand and address the “fuzzy boundary” of what the model should be able to do [00:24:01].