From: aidotengineer
This article, drawn from an inverted presentation on how to “spectacularly mess up your AI strategy,” highlights critical missteps to avoid when developing and deploying AI systems [00:00:27]. By understanding these “worse practices,” organizations can navigate the complexities of AI development more effectively [00:00:54].
Fostering Internal Division and Disconnect
A primary pitfall is the deliberate creation of internal silos and a disconnect within the company [02:27:07].
Siloing Knowledge
- Lack of Communication Attending AI industry conferences but never sharing learned insights with the team creates “impenetrable silos” and incentivizes secrecy [02:53:05]. The goal is to divide and conquer one’s own company [02:27:07].
- Embracing Disconnect A key mistake is to disregard the relationship between willingness to pay, price, and cost, aiming for “unreasonable goals” that fail to create value [02:35:08]. This is framed as adhering to an “anti-value stick,” the opposite of strategic value creation [03:18:05].
Mismanaging Value Perception
- Wishful Thinking Promises (WTP) Making exaggerated promises to customers about AI’s capabilities, such as solving climate change or achieving world peace, without worrying about the details [03:32:05].
- Particularly Ridiculous Infrastructure Costs (PRICE) Buying the most expensive GPUs without cost-benefit analysis, maxing out company credit cards as a vague “investment” [03:54:05].
- Cascade of Spectacular Technical Debt (COST) Building convoluted, intertwined systems that even executives barely understand, leading to guaranteed job security when they inevitably break [04:16:05].
- Why This System? (WTS) Justifying AI projects solely with “because AI” without further explanation, treating it as “magic but much more expensive and less reliable” [04:45:05].
Defining a Flawed AI Strategy
Poor strategy definition is a guaranteed path to failure [05:04:05].
Vague and Unrealistic Objectives
- Fake Diagnosis Grabbing random, poorly understood paragraphs from old reports and declaring them “must fix” issues without consulting those who do the actual work [05:06:05].
- Ambiguous Guiding Policy Defining a strategy that is “incredibly ambiguous and vague,” such as aiming to “become the global AI leader in everything” without defining what “everything” means [05:27:05].
- Absurd Action Plans Including nonsensical action items like an “AI-powered SEO tool that guarantees top Google search results even if you sell garden gnomes,” a generative art plug-in for NFTs of the CEO’s cat, or an “AI drone lunch delivery service” [05:41:05].
- Ignoring Timelines Embracing “Perpetual beta” and never intending to finish projects, instead creating massive backlogs with highlighted financial reports [06:07:05].
- Overwhelming Documentation Creating overly long documents (e.g., 4,000 pages) and distributing them widely to “erode people’s willpower to engage with the material” [06:25:05].
Communication Breakdown and Jargon Abuse
One of the most effective ways to cause dysfunction is through pervasive and strategic use of jargon [07:31:05]. This is a common communication challenge in AI deployment.
- Obfuscation Using jargon to appear smart without being understood, making it impossible for others to engage or understand the true purpose [06:46:05]. An example is “our multimodal agentic Transformer-based system leverages few-shot learning and Chain of Thought reasoning to optimize the synergistic potential of our Dynamic hyperparameter space” [06:53:05].
- Hiding “Jobs to Be Done” Using technical terms like “building agents” instead of “writing a prompt” to exclude non-technical experts, such as mental health professionals, from understanding or participating in the process [07:41:05]. This contributes to design challenges for AI agents.
- Excluding Domain Experts Using terms like “RAGs” instead of “making sure the AI has the right context” or “prompt injections” instead of “making sure users can’t trick the AI into doing something bad” to alienate and exclude domain experts [08:12:05].
- Engineer-Centric Prompting Encouraging engineers, rather than those who best understand customers, to write prompts [08:24:05].
- Making Everything Seem Technical Deliberately making mundane tasks, like writing prompts, appear “super technical and out of reach for everyone” [08:50:05].
Ineffective Mobilization of Teams
Mobilizing teams incorrectly is a significant pitfall that can lead to total collapse [09:08:05].
Misallocating Resources
- Random Task Assignment Randomly assigning AI tasks to individuals with “absolutely no relevant experience” [09:24:05]. This is a clear case of mismanagement of AI resources.
- Outsourcing Without Context Outsourcing critical tasks like data review to offshore Q&A teams with “very little context about your business” [09:32:05].
- Premature Launching Launching “completely untested bug-ridden AI chatbots directly to your customers” from an “incubation zone” without beta testing or quality assurance [09:41:05]. This is a common challenge with current AI implementation.
- Disrupting Core Business Yanking the best engineers from revenue-producing products to work on AI projects, leading to “total collapse” [10:12:05].
Over-reliance on Tools and Data Avoidance
A critical mistake is to prioritize tools over processes and actively avoid looking at data [10:39:05].
Tool-Centric Problem Solving
- Ignoring Process Focusing on tools instead of processes to solve problems, throwing new tools at issues without analyzing or understanding them [10:42:05].
- Blindly Adopting Tools For instance, if a RAG system fails, immediately buying a new, more expensive vector database rather than diagnosing the root cause [10:51:05].
- Uncritical Evaluation Using every “off-the-shelf evaluation metric” without customizing them to business needs, blindly trusting numbers even if they make no sense [11:01:05]. This contributes to challenges in building reliable AI agents.
- Frequent Framework Changes If AI agents aren’t working, picking a new framework and vendor, or fine-tuning without any measurement or evaluation, assuming it will be better [11:15:05]. This highlights common development challenges for AI agents.
- “Whack-a-Mole” Approach Consistently hammering every problem with a new tool, even if the same problem recurs [11:46:05]. This is the opposite of best practices for building AI systems.
- “Vendor Problem” Mindset Believing that evaluations are solely a “vendor problem” and that a “one-size-fits-all solution” exists, allowing vendors to figure everything out [12:09:05]. This also impacts cost-effective AI strategies.
Misleading Metrics and Data Avoidance
- Meaningless Dashboards Creating dashboards filled with numerous off-the-shelf metrics that don’t track with outcomes or real failure modes, making numbers unintelligible [12:26:05].
- Hoarding Random Metrics Hoarding random metrics until one goes “up and to the right,” then claiming success [12:44:05]. This is a common pitfall in AI agent development.
- Blindly Trusting Metrics Adopting all metrics from eval frameworks blindly and never asking whether they actually measure success [13:01:05].
- Ignoring User Experience Optimizing for metrics like cosine similarity, BLEU, and ROUGE while ignoring actual user experience [13:17:05].
- Avoiding Data Actively avoiding looking at data, keeping a “blindfold” ready to avoid accidental data exposure [13:42:05].
- Implicit Trust in AI Absolutely trusting AI output without ever looking at it, considering data analysis an “engineering problem” [13:56:05].
- Ignoring Customers Believing developers have more domain expertise than business teams and treating customers as the “best Q&A” [14:18:05].
- Trusting Gut Over Data Relying on gut feelings as a “reliable substitute for data” when making million-dollar decisions [14:34:05].
- Complex Data Storage Storing data in complex systems accessible only to engineers, thereby preventing domain experts from accessing or understanding it [15:20:05]. Insisting on expensive custom data analysis platforms requiring PhD teams to operate, taking months to load and prone to errors [15:39:05].
By consciously avoiding these “worst practices,” organizations can prevent wasting time and resources, and foster better collaboration within their teams [16:04:05].