What is a Backtest or Batch Re-run?
At Alloy, you can make two types of batch requests: Backtest or Batch re-run. Backtesting or Batch Re-running is a predictive modeling method that uses historical data to simulate and test the efficiency of a given model, or in this case, the workflow.
Purpose of a Batch Request
A Backtest is when Alloy runs a batch of real PII data of applicants (we cannot backtest with fake data) that has not been sent to Alloy before. Alloy will provide the CSV template for the organization to fill out. In the CSV file, the organization will provide PII to run through a specific workflow to see how it performs. A Backtest creates new entity tokens because the evaluations have not yet been run through Alloy.
While a Backtest runs real PII data that has not been sent to Alloy before and provided by the organizations, a Batch Re-run is when previously run evaluations (in Alloy) are re-run against a new workflow or workflow version. The organization provides Alloy with a list of evaluation tokens to re-run.
Since a Batch Re-run is performed on entity data already within Alloy, we use the same data to see how the workflow performs. A Batch Re-run does not create new entity tokens because the evaluations will be linked to the existing entities unless otherwise specified.
Both batch requests allow organizations to test historical client data against a workflow to understand better what rules should be tweaked to align with the organization’s internal policy or procedural needs. These batch runs allow organizations to compare past outcomes against the decision engine before going live (e.g., a Backtest will enable clients to validate their workflow within Alloy).
When running these batch requests, organizations already know what changes they need to make (e.g., based on a change to their credit policy). Alloy will translate those changes into the workflow and then make sure the organization is getting the expected output.
Note: There may be costs associated when running one of these batch requests based on the entity and service fees set in your contract.
Backtest/Batch Re-run vs. What If Analysis
Slightly different from a Backtest or Batch Re-run, a What If Analysis is a feature that allows organizations to safely experiment with making updates and changes within the workflow to see if they can drive better outcome results using historical data. Ultimately, a What If Analysis is done when the organization wants to answer, “did this change work, or did it break something?”
To learn more about What If Analysis, please visit Using What If Analysis. Things to note on What If Analysis are the following:
- Does not work with an additional data source(s) (e.g., can’t add a new data service to your workflow and then run a What If Analysis)
- Can remove existing data source(s)
- Runs up to 500 random evaluations
Please don’t hesitate to contact support@alloy.com or your dedicated CSM to see if a backtest or batch re-run request is right for you.
Comments
0 comments
Article is closed for comments.