Hello everyone, Greg here, the Head of AI and Innovation at 6clicks. Today we're going to explore how we test, train, and refine the foundational AI model behind our Hailey Assist engine.
Standardized testing process
We rely heavily on a standardized set of questions or prompts to evaluate the performance and quality of responses of Hailey Assist. This internal, crowd-sourced list of prompts encompasses a broad range of queries that users might ask. By consistently using this set, we can effectively measure the engine’s performance against a controlled dataset that remains unchanged. This consistency is crucial for accurately assessing whether our engineering efforts are improving the model or not.
Response evaluation criteria
Each response generated by Hailey Assist is evaluated based on both the content of the answer and the links or hyperlinks provided. We use a three-tiered scoring system:
- Score of three (Great): The response perfectly answers the question. It’s the ideal outcome where no improvements are necessary.
- Score of two (Passable): The response is reasonable and technically answers the question or points towards the answer, even if it’s just through a link.
- Score of one (Needs improvement): The response fails to meet expectations. It might be the wrong answer, an incorrect link, or an error due to insufficient knowledge or coverage in the database.
Continuous improvement
With each new release of Hailey Assist, we run these standardized tests to track and tune the AI’s performance. This rigorous testing process ensures that we continually refine the model, making sure it provides accurate and helpful responses over time.
This process gives you a glimpse into the meticulous work that goes into building and tuning the Hailey engine. It’s an ongoing effort to ensure that our AI model meets the high standards we set for providing reliable and precise assistance.
See Hailey Assist in action
Here's a short video to help you better understand just what goes into the process of testing and training our AI engine:
Stay tuned for more updates on how we’re enhancing Hailey Assist.
Written by Greg Rudakov
Greg is a Senior Product Manager at 6clicks, spearheading the growth and development of the company's groundbreaking Hailey AI engine. Greg's impressive track record includes founding a successful SaaS venture, leading major projects for companies such as KPMG, and integrating IT systems and teams across the globe. With a keen focus on go-to-market strategy and collaboration amongst senior leadership and customers alike, Greg continues to elevate 6clicks' position as a leader in AI-driven solutions for risk management and compliance.
