Exploring the AI model testing and training process for Hailey Assist

Hello everyone, Greg here, the Head of AI and Innovation at 6clicks. Today we're going to explore how we test, train, and refine the foundational AI model behind our Hailey Assist engine.

Standardized testing process

We rely heavily on a standardized set of questions or prompts to evaluate the performance and quality of responses of Hailey Assist. This internal, crowd-sourced list of prompts encompasses a broad range of queries that users might ask. By consistently using this set, we can effectively measure the engine’s performance against a controlled dataset that remains unchanged. This consistency is crucial for accurately assessing whether our engineering efforts are improving the model or not.

Response evaluation criteria

Each response generated by Hailey Assist is evaluated based on both the content of the answer and the links or hyperlinks provided. We use a three-tiered scoring system:

1. Score of three (Great): The response perfectly answers the question. It’s the ideal outcome where no improvements are necessary.
2. Score of two (Passable): The response is reasonable and technically answers the question or points towards the answer, even if it’s just through a link.
3. Score of one (Needs improvement): The response fails to meet expectations. It might be the wrong answer, an incorrect link, or an error due to insufficient knowledge or coverage in the database.

Continuous improvement

With each new release of Hailey Assist, we run these standardized tests to track and tune the AI’s performance. This rigorous testing process ensures that we continually refine the model, making sure it provides accurate and helpful responses over time.

This process gives you a glimpse into the meticulous work that goes into building and tuning the Hailey engine. It’s an ongoing effort to ensure that our AI model meets the high standards we set for providing reliable and precise assistance.

See Hailey Assist in action

Here's a short video to help you better understand just what goes into the process of testing and training our AI engine:

Stay tuned for more updates on how we’re enhancing Hailey Assist.