From: redpointai
When evaluating AI applications, it’s recommended to start small and gradually increase complexity, always justifying progress with a rigorous return on investment (ROI) [00:00:00]. The core question is whether the AI is making progress on what truly matters to the user [00:00:05].
The Iterative Process of Benchmarking
A crucial step in this process is to build some benchmarks and test against them [00:00:08]. It’s common to discover that initial benchmarks are insufficient, necessitating the creation of better ones [00:00:09]. This iterative approach is vital for effective AI evaluation.
Starting Your AI Experiment
The journey can begin with minimal investment, perhaps spending as little as 20 cents on platforms like OpenAI or Llama on Databricks, to litmus test whether AI is suitable for a specific task [00:00:12]. There is currently a lack of predictability regarding AI’s effectiveness for particular use cases [00:00:18].
It’s advised to approach this like a scientist engaging in data science: run an experiment [00:00:23]. To maximize the chance of success, set up the experiment carefully [00:00:30] and utilize the best available model [00:00:31].
Initial testing can be as simple as:
- Prompting the model [00:00:34].
- Manually providing a few relevant documents into the context, without relying on Retrieval-Augmented Generation (RAG), to observe the outcome [00:00:34].
The goal is to determine if there’s any value or “there there” before scaling up [00:00:41].
Advanced Steps: RAG and Fine-Tuning
If initial tests show promise, the next step might involve implementing more sophisticated RAG techniques to integrate proprietary data, as models do not inherently possess knowledge of internal enterprise information [00:00:45].
Further down the line, if significant value is being generated, fine-tuning models becomes a consideration [00:00:53]. While fine-tuning incurs a greater upfront cost, it can embed learned patterns directly into the model, leading to improved quality outcomes [00:00:54].