Real estate due diligence automation using AI

From: aidotengineer

Orbital is a company with offices in New York and London, whose mission is to automate real estate due diligence to accelerate property transactions [02:01:06]. This process is critical because homes and offices are constantly developed, bought, and sold, involving real estate lawyers in due diligence [02:08:46].

The Problem: Manual Due Diligence

Traditionally, real estate lawyers perform due diligence by reading “mountains of paperwork” and “hunting for needles in a haystack” to identify red flags for clients before transactions can proceed [02:17:40]. This manual process is time-consuming and inefficient [02:19:12].

The Solution: Orbital Copilot

Orbital developed an agentic system called Orbital Copilot, launched in January 2024, which “thinks like a real estate lawyer” [03:17:34]. This Agentic software significantly reduces the time required to find critical information and compile necessary paperwork for real estate transactions [02:29:08].

How it Works

The Orbital Copilot demo illustrates the automation of a long-running task previously performed manually by lawyers [03:46:04]:

1. Report Selection and Document Upload [04:01:00]: Users select a report type (e.g., occupational lease) and upload documents (e.g., deed, lease) [04:04:00].
2. OCR and Structuring [04:17:28]: The system first performs Optical Character Recognition (OCR) on documents containing handwritten and typed text to structure the data [04:20:00].
3. Agentic Task Execution [04:27:00]:
   • The agentic system creates a plan, breaking it into many subtasks [04:30:00].
   • Each subtask is its own agentic system involving multiple LLM calls [04:36:00].
   • The system is given objectives, such as finding the lease date or current annual rent [04:41:00].
   • It reads legal documents to find appropriate answers [04:46:00].
4. Report Generation and Review [04:56:00]: Once all subtasks are complete, a final report is generated for manual review by a lawyer [04:58:00]. Citations can be clicked to go back to the original ground truth [05:07:00].
5. Download and Client Delivery [05:15:00]: The final word report can be downloaded, stored, and sent to a client to progress the transaction [05:17:00].

Impact and Results

Since commercializing its agentic agent 18 months ago, Orbital has seen significant growth [05:34:00]:

• Token Consumption: From burning less than a billion tokens monthly to consuming almost 20 billion tokens every month on behalf of real estate lawyers [05:39:00]. This represents 20 billion tokens worth of work previously done manually [05:55:00].
• Revenue Growth: From zero revenue to multiple seven figures in annual recurring revenue, which continues to scale [06:06:00].

Technical Approach and Challenges

Orbital’s journey involved migrating through various LLM models, starting from GPT-3.5 and moving through GPT-4 32K, 4 Turbo 40, 4.1, and system 2 models like 01 preview to 04 mini [06:28:00].

Key Decisions

1. Optimize for Prompting over Fine-tuning: This approach maximized development speed, allowing real-time adjustments to prompts based on user feedback to quickly incorporate changes and find product-market fit [07:00:00].
2. Heavy Reliance on Domain Experts: Private practice real estate lawyers, with decades of experience, are embedded in the team and write many prompts [07:34:00]. They effectively teach the AI system their expertise [07:49:00].
3. “Vibes over Evals”: While an evaluation system is on the roadmap, Orbital has achieved significant growth in tokens, revenue, and user feedback largely based on subjective human testing by domain experts before release [08:01:00]. This involves subjective feel, logging regressions in spreadsheets, but nothing “terribly comprehensive” [08:29:00].

Prompt Taxonomy

Prompts are categorized into two areas [08:52:00]:

• Agentic Prompts: Owned by AI engineers, these are system prompts that help the model choose which tools to use and when [08:56:00].
• Domain-Specific Prompts: Used by real estate lawyers to impart real estate expertise to the system [09:09:00]. The number of these prompts has grown from near zero to over 1,000 [09:21:00].

The “Prompt Tax”

The increase in prompts leads to a “prompt tax” [09:30:00]. When a new AI model is released, Orbital rigorously experiments with it [09:39:00]. This involves:

• Unlocking envisioned features with new capabilities [09:50:00].
• Assessing the “prompt tax” required to migrate existing prompts [10:04:00].
• Dealing with inherent fear due to unknown unknowns when shipping a new AI model [10:10:00].

The “prompt tax” is distinct from technical debt; it’s the cost of upgrading to new models that offer new capabilities but also introduce uncertainty about what will improve or break [10:59:00].

Battle-Tested Tactics

Orbital has developed several tactics for navigating the rapidly evolving AI landscape [11:52:00]:

• Adapting Prompts for System 2 Models: For newer, more capable models (System 2 like 01 preview), prompts need to be less specific, leaner, and avoid repeating instructions, focusing on clearly stating what to do rather than how [12:12:00]. They also benefit from being “unblocked” with clear objectives and time to reason [12:40:00].
• Utilizing Thought Tokens: Though System 1 models are often cheaper and faster, their thought tokens can be embedded for user explainability (especially useful for real estate lawyers needing to understand complex legal reasoning) or for debugging [13:07:00].
• Progressive Rollout with Feature Flags: Similar to software development, new AI model upgrades can be rolled out progressively to mitigate risk, though “change aversion bias” can heighten anxiety [13:46:00].
• “Betting on the Model”: The team mantra is to anticipate future AI model advancements (smarter, cheaper, faster, more capabilities) and build features that will improve as models become more capable [14:56:00].
• Using System 2 Models for Prompt Migration: Newer models, being inherently more capable, can assist in migrating older domain-specific prompts, significantly reducing manual human effort [15:45:00].
• Decisive Shipping: Given the probabilistic nature and uncertainty of new AI models, teams need to be brave enough to ship and deal with consequences, mitigating risks along the way [16:11:00].
• Strong Feedback Loops: Rapid feedback from users (manual or via in-product UX like thumbs up/down) is crucial [17:10:00]. This feedback is sent to AI engineers and domain experts, allowing prompt changes and production deployments in minutes or hours [17:24:00].

The Challenge of Rapid Technological Evolution

Deis Havaris, Chief Exec of Google DeepMind, highlights that the underlying AI tech stack is evolving “unbelievably fast,” unlike previous revolutionary technologies [18:17:00]. This poses a unique challenge for product development, as companies must decide what to “bet on” when the technology could be “100% better in a year” [18:55:00]. This requires deeply technical product people who understand where the technology might go to design future-proof products [19:06:00].

Paying the Prompt Tax: Future Considerations

The meta-question for agentic product developers is how to gain more confidence when shipping at the AI frontier [20:38:00]. While Orbital has scaled successfully on “vibes” (human review and rapid feedback), it’s uncertain if this will scale indefinitely as the product surface area grows [21:03:00].

The possibility of an evaluation (eval) system is considered [21:32:00]. However, the complexity of evaluating LLM outputs in real estate legal contexts (correctness, style, conciseness, citation accuracy, edge cases vs. happy paths) makes building a comprehensive eval system prohibitively expensive and slow, possibly an impossible task given rapid product velocity [21:49:00].

Progressive Delivery

A potential way forward is progressive delivery, rolling out new models internally first, then to a limited number of users, and incrementally scaling up based on feedback [22:30:00]. This allows for “fixing on the fly” [22:39:00].

Conclusion

To stay at the AI frontier and maximize opportunities from new model capabilities, it’s essential to ship products and get them into users’ hands quickly [23:37:00]. The “prompt tax” (the cost of adapting to new models) and anxiety over potential downsides may not always materialize, or can be managed through strategies like incremental rollouts and rapid feedback loops [24:00:00].