From: allin
Ilya Sutskever, a co-founder and former Chief Scientist at OpenAI, has launched a new startup named Safe Superintelligence Inc (SSI) [01:14:14].
Formation and Background
Sutskever resigned from OpenAI in May 2024, after a decade with the company where he also served as co-head of super alignment [01:14:25]. His departure followed a period of internal turmoil at OpenAI, where he was part of the board that attempted to oust CEO Sam Altman in November 2023, only to reverse course days later and express regret [01:14:31].
The co-founders of SSI include Daniel Gross, a YC partner and Pioneer Labs co-founder, and Daniel Levy, an OpenAI engineer [01:14:44].
Mission and Strategy
The company’s primary goal is stated within its name: to develop a safe superintelligence [01:14:52]. Sutskever explicitly stated that SSI’s “first product will be the safe super intelligence and it will not do anything else until then” [01:14:57].
Industry Reaction and Challenges
Commentators have expressed mixed reactions and identified several challenges for SSI’s unique approach:
- “Throttle” or “Governor” on Development: The focus on “safety” is seen by some as a potential hindrance, akin to a “throttle” or “governor” on a startup that needs to move quickly to compete [01:55:58]. This focus on safety concerns acts as a “brake pedal” and can slow down development [01:56:19].
- Competitive Disadvantage: It has been suggested that OpenAI CEO Sam Altman may have intentionally limited resources for safety-focused groups within OpenAI to prioritize speed in AI advancements [01:56:30]. This competitive landscape suggests that “companies that care about safety more than others are going to lose” in the race to achieve Artificial General Intelligence (AGI) [01:57:42].
- Investment Requirements: The cost of developing functional AI models is rapidly increasing, from billions of dollars today to potentially hundreds of billions by 2027 [01:00:01]. This creates an “arms race on cost and compute” [01:01:06]. Securing such massive funding could be a significant challenge for a new startup, as major tech companies like Google, Microsoft, Facebook, and Amazon have deeper pockets to invest in this area [01:00:14].
- Market Dynamics: Foundational models are quickly becoming a “consumer surplus,” with models approaching asymptotic returns [00:59:20]. This implies that differentiation might require increasingly expensive and refined data processing, leading to market conditions where “one startup can probably win but there will be a bunch of open source alternatives” that are “asymptotically similar” [01:01:00].