From: aidotengineer
Domain experts play a crucial role in the development and refinement of AI systems, particularly at companies like Orbital, which focuses on automating real estate due diligence [02:01:43]. Their specialized knowledge is vital for teaching AI models, evaluating performance, and ensuring the practical applicability of the AI solutions [07:51:00].
Integration into Engineering Teams
At Orbital, the product engineering team, which is about half of their 80-person staff, includes embedded domain experts alongside product managers, designers, software engineers, and AI engineers [02:47:00].
Composition of an AI Product Team
An effective AI product team at Orbital is structured with a product manager and designer, embedded domain experts, software engineers, AI engineers, and a tech lead [02:51:00]. This cross-functional setup ensures the delivery of functional agentic products [03:06:00].
Key Responsibilities and Contributions
Prompt Engineering and Expertise Transfer
A core responsibility of domain experts at Orbital is to write the prompts that infuse the AI system with their specialized knowledge [07:45:00].
Real Estate Lawyers as Domain Experts
Orbital heavily relies on private practice real estate lawyers who have decades of experience [07:34:00]. These experts effectively “teach” the AI system their legal expertise by writing domain-specific prompts [07:47:00]. The number of these domain-specific prompts has grown from nearly zero to over 1,000 [09:18:00].
Evaluation and Feedback Loops
Domain experts are instrumental in the evaluation and feedback process, particularly given the dynamic nature of AI model development [08:06:00].
- Testing and Validation: They are involved in rigorously experimenting with new AI models as they are released [09:39:00].
- Subjective “Vibes” over Formal Evals: In the absence of a comprehensive, objective evaluation system, Orbital has relied on “vibes,” meaning human evaluation by domain experts who test the system before release [08:01:00]. This involves subjective assessment and sometimes logging issues in spreadsheets to identify regressions from prior model changes [08:36:00].
- Rapid Feedback Integration: Domain experts receive user feedback directly through the product, evaluate it, identify which prompt needs modification, and then implement that change into production, often within minutes or hours [17:27:00]. This rapid cycle allows for quick fixes and continuous product improvement [17:48:00].
Challenges and Future Outlook
The increasing number of prompts introduces a “prompt tax,” making it more challenging to manage as the system grows [09:30:00]. Despite the success achieved through a combination of domain expert input and user feedback, there’s an acknowledgment that relying solely on “vibes” may not scale indefinitely as the product’s surface area expands [21:06:00]. The complexity of evaluating aspects like correctness, style, conciseness, and citation accuracy in legal AI makes creating a comprehensive evaluation system a challenging task [21:49:00].