Analytics
This guide will go over the basics of using the bottest.ai Analytics Page (opens in a new tab), including how they are gathered and what each chart means. You can always navigate to the Analytics Page by clicking "Analytics" in the menu on the left. Note that all Analytics are for Suite Runs only. Individual Test Runs will not show up in this data.
Viewing Analytics
You can manage which Bot's/Suite's/Environment's analytics will be shown using the dropdown menus:
All Tests belong to a Suite, and all Suites belong to a Bot. If you're unsure how you should organize your Tests into Suites and Bots, check out the documentation on Bots, Suites, and Environments.
Understanding Each Chart
This next section will go over each chart in the Analytics section and explain what is being measured, and the possible implications or insights to look out for with each one.
Success
The Success category has 3 charts:
- Evaluations Performed. This chart measures the number of total evaluations performed in the last 10 Suite Runs.
You are recommended to keep and eye on this chart to monitor the size of your Test Suite as it grows and new Tests/Variants are added or iteration count is increased. Additionally, this chart gives context to some of the other charts. For example, if your last 10 Suite Runs have had a trend of increasing number of evaluations while the evaluation pass rates have been declining, it may be the case that only the new Tests added are failing. However, if the evaluations performed is consistent and pass rates are lowering, it means there is some level of regression happening in the chatbot performance.
- Test Statuses. This chart measures the Test Status percentages for the last 10 Suite Runs.
You are recommended to watch this chart to see the trend of how Tests are performing over time. You will want to watch to make sure that Pass rates are increasing, while Fail rates are either turning into Mixed or Pass. This helps gives insight into the performance of your Tests on a high level.
- Evaluation Pass Rate. This chart measures the total evaluation pass rates for the last 10 Suite Runs.
While this is similar to the Test Statuses, this gives the actual underlying evaluation pass rate across all Tests in the Suite Runs. This chart is especially useful if the previous chart contains a lot of Mixed statuses. It's hard to know if those Mixed are 90% passed with 10% failed, or 90% failed with 10% passed. This chart helps give important insight into the overall performance of your Tests across all evaluations.
Performance
The Performance category has 1 chart to measure the Test execution time for the last 10 Suite Runs. This chart provides a box plot of the distribution of all execution times (the time it takes for the each conversation to happen on the webpage).
This is helpful for tracking the response time from your chatbot on a high level, and see if your chatbot's performance is trending worse.
Usage
The Usage category has 2 sections to help your team manage your usage of the bottest.ai Platform:
- Test Runs Performed in the last month. This chart breaks down the total Test Runs performed during Suite Runs and individually on a day-by-day basis for the last month (for the selected Suite + Environment).
You should use this data to understand on which days Test Runs are ran, and the extent to your usage for the selected Suite and Environment.
- Plan Information and Usage. This widget shows your current to-date usage for the pricing plan that is selected. You can keep an eye on this tracker to see if your team might want to upgrade or downgrade in the pricing plan.
