Open the Testing Suite
Before opening the Testing Suite, you must first apply your changes to a Workflow version that has received at least one evaluation in the production environment.
Once you make an (unsaved) change to your Workflow in the Workflow Editor, you’ll see a Test button with a red blinking badge on the top right.
Click on this Test button to open the Testing Suite:
If you notice the Test button is grayed out, it may not meet the requirements to open the Testing Suite. Hovering over the button will display an error message to explain the reason.
| Error Message | Reason |
| Make a change to the Workflow to test its impact. |
No changes have been made to the version. Make at least one change to the Workflow that What If Analysis supports testing. |
| This Workflow version is not eligible for What If Analysis. |
There have been no evaluations sent to the Workflow version being edited. Send at least one production evaluation to this version, or edit a version that has received at least one production evaluation. |
Preview Changes
The Preview Changes section is where you can see the changes that will be tested against your saved version:
Click Next once you have finished reviewing, or click Back to Workflow to return to the Workflow and continue making changes.
Select Test Type
From the Select Test Type section, select the What If Analysis option.
Then, select Yes or No for the question Do you want to generate new test evaluations and save this Workflow Version?
We recommend Yes, but you can try out both!
| Option | Definition |
| Yes |
Alloy will save your changes to a new inactive Workflow version. Alloy will generate new evaluations from your testing sample, which will be stored and accessible for review anytime on the Test History page. |
| No |
The View Results step will only show the result summary of the test. You will not be able to review which evaluations were used for the What If Analysis test. |
Click Next when ready to move to the next step, or you can click Previous to return to the previous step at any time.
Filter Test Sample
Configure your testing sample to test your changes against in the Filter Test Sample section.
Select which test sample you’d like to use, and configure it further by choosing Random Evaluations or Specific Evaluations:
- Random Evaluations - Alloy selects random evaluations based on the specified evaluation created date, timeframe, and sample size.
- Specific Evaluations - You can more granularly select which random evaluations based upon the inputted evaluation create date and sample size.
| Filtering Options | Definition | Random Evaluation | Specific Evaluations |
| Created At |
Choose the exact timeframe from the calendar or choose an option from one of the quick select options: Last 30 minutes, Last 1 hour, Last 3 hours, Last 6 hours, Last 12 hours, Last 24 hours, Last 3 days, Last 7 days, Last 30 days, Last 3 months, Last 6 months, Last 1 year, Today, Yesterday, This week, This month, This quarter, This year |
✅ | ✅ |
| Sample Size |
Select a minimum of 1 sample or a max of 5000. For statistical significance, we recommend a minimum sample size of 500. |
✅ | ✅ |
| Search Evaluations | Add a comma-separated list of evaluation tokens that you want to test. (Example: L-86WFI55CVlESLCpMQmVZ, L-F46yo1ATr14arAUeVPYG) |
❌ | ✅ |
| Outcome |
Select which outcomes you’d like. If any value is added to the Search Evaluations option, this option will not be available. |
❌ | ✅ |
| Tags | If any value is added to the Search Evaluations option, this option will not be available. | ❌ | ✅ |
Click Next when ready to move to the next step.
View Results
|
💡IMPORTANT💡: Please note that no evaluations are being rerun, and getting new data for the evaluations. What If Analysis uses cached data from the existing evaluations and applies the workflow changes to those preexisting data to determine how the outcome rate would change. |
The View Results page contains the Result Summary section that shows a bar graph of the original version’s outcomes vs. the saved/created test workflow version’s outcomes.
If Yes was selected in the Select Test Type section, an Evaluations section will also be displayed.
To view individual evaluation results of an entity, click on the evaluations.
Once you click Finish, Alloy will navigate you back to the Workflow version you were editing and retain the changes you made if you want to keep editing.
Comments
0 comments
Article is closed for comments.