Excel-based tool enables business analysts to create analytics-powered decision models, “point and click” without coding, using optimization, Monte Carlo simulation, machine learning, and business rules; now includes innovative, fully automated methods for assessing risk of unexpected results from machine learning models when deployed for production use.
INCLINE VILLAGE, NV – September 18, 2022 – Frontline Systems is shipping Analytic Solver® V2023, a new version of its advanced analytics toolset for Excel (Web, Windows and Macintosh), that enables business analysts to easily build models using business rules, machine learning, mathematical optimization and Monte Carlo simulation, and easily deploy those models in cloud-based applications.
Analytic Solver is not new – it’s a market-leading analytics tool upward compatible from the Solver in Excel, which Frontline originally developed for Microsoft. But now it’s the first and only tool with fully automated methods for risk analysis of previously trained and validated machine learning (ML) models.
As an “Excel Solver upgrade”, Analytic Solver can handle virtually any type or size of optimization problem, ranging from a few to millions of inter-related decisions in a single model. And for years, Analytic Solver has offered powerful features for risk analysis using Monte Carlo simulation, and powerful features for training, validating, and deploying predictive models using machine learning.
Now, Analytic Solver V2023 includes a innovative capability for risk analysis of machine learning models that leverages multiple capabilities of the software. Risk analysis changes the focus from how accurately a ML model will predict a single new case, to how it will perform in aggregate over thousands or millions of new cases, what the business consequences will be, and the (quantified) risk that this will be different than expected from the ML model’s training and validation.
“With a patent application now on file to preserve invention rights, Analytic Solver users are the first to benefit from these innovative methods”, said Daniel Fylstra, Frontline’s President and CEO. Frontline Systems is concurrently releasing new versions of RASON®, its cloud platform for analytics, and Solver SDK®, its toolkit for software developers, with support for the same innovative methods.
How and Why Machine Learning has Lacked Risk Analysis
For a decade, data science and machine learning (DSML) tools – including Analytic Solver – have offered facilities for ‘training’ a model on one set of data, ‘validating’ its performance on another set of data, and ‘testing’ it versus other ML models on a third set of data. But this is not risk analysis: based on known data, it doesn’t assess the risk that the ML model will perform differently on new data when put into production use. While it’s common to assess a ML model’s performance in use, and move to re-train the model if its performance is unexpectedly poor, by that time those risks have occurred, often with adverse financial consequences. Quantification of such risks “ahead of time” has been missing in practice.
There are many reasons for this state of affairs: Data scientists with expertise in ML methods often are not trained in risk analysis; they think of “features” and even predicted output values as data, not as “random variables” with sampled instances. Even if known, conventional risk analysis methods are expensive and time-consuming to apply to machine learning: ML data sets include many (sometimes hundreds) of features, with limited “provenance” of the data’s origins. There are hundreds of classical probability distributions that could be ‘candidates’ to fit each feature. Only some of the features are typically found, after ML model training, to have predictive value; many are found to be correlated with other features and hence ‘redundant’. And in typical projects, a great many ML models are built.
How Analytic Solver Performs Automated Risk Analysis
Unlike most other DSML software, Analytic Solver includes powerful algorithms for risk analysis in the same package: Probability distribution fitting, correlation fitting, stratified sample generation and Monte Carlo analysis. But asking business analysts – let alone data scientists – to “quickly master risk analysis” is asking too much. So Frontline Systems has invented ways to automate the entire risk analysis process. Using the new capability is as simple as checking one extra box in a dialog, with some further “point and click” options for analysis and reporting – and the risk analysis typically adds just seconds to a minute to the existing process of training, validating and testing a ML model.
Behind the scenes, for each feature, Analytic Solver tests an entire family of probability distributions – drawing on its first-mover support for the new Metalog family of distributions, created by Dr. Tom Keelin; optimizes all the parameters of each distribution; detects and models correlations among features, using rank order and copula methods; performs synthetic data generation, using Monte Carlo methods for stratified sampling and correlation; computes the ML model’s predictions, as well as user-specified financial consequences, for each simulated case; and importantly, assesses and quantifies the differences in performance of the ML model on this simulated data versus the training, validation and test data.
The user sees results of the risk analysis in automatically-generated charts, statistics and risk measures – drawing from the Monte Carlo simulation features of Analytic Solver, proven in use over many years.
Synthetic Data Generation as a Side Benefit
Synthetic Data Generation (SDG) has become topical in machine learning in recent years, with a number of companies founded just to supply software and services around this technology. SDG is used when there isn’t enough original data, or when use of the original data is restricted by law or regulation. But until now (in a patent and literature search), SDG has simply been used to better train ML models.
Analytic Solver V2023 includes a powerful, general-purpose, easy to use Synthetic Data Generation facility, accessible from the Excel Ribbon. Unlike some special-purpose SDG offerings, this facility can accurately model the behavior of nearly any combination of features with continuous values. But Analytic Solver also uses synthetic data in an entirely new way, to analyze the risk that a ML model will yield unexpected results “large enough to matter” when deployed for production use.
Works with Already-Available ‘Augmented Machine Learning’
Analytic Solver’s V2021.5 release featured “augmented machine learning” features found only in other sophisticated machine learning tools. The user simply supplies data, and selects a menu option “Find Best Model”: Analytic Solver will automatically test and fit parameters for multiple types of machine learning models – classification and regression trees, neural networks, linear and logistic regression, discriminant analysis, naïve Bayes, k-nearest neighbors and more – then validate and compare them according to user-chosen criteria, and deliver the model that best fits the data.
Analytic Solver V2021.5 also featured enhancements that enable multi-stage “data science workflows” including machine learning models built and tested in Excel, to be deployed automatically to RASON® Decision Services, Frontline Systems’ comprehensive cloud platform for decision intelligence.
But now that it’s possible and even easy, users will want to assess the risk of a ML model before it is deployed. With a few mouse clicks, Analytic Solver V2023’s automated risk analysis can be applied to the model delivered by “Find Best Model”. When the analyst or decision-maker is satisfied, with a few more mouse clicks, the model can be deployed as a cloud service with an easy-to-use REST API.
Free Trials, Learning and Coaching Resources
Business analysts and developers can sign up for free trial accounts to evaluate Analytic Solver at https://www.solver.com, and RASON at https://rason.com. They can use tools to create and solve models in Excel and RASON, exercise the REST API, try out dozens of example models illustrating use of decision tables, predictive models and machine learning, optimization and simulation, and download the Analytic Solver and RASON User Guides and Reference Guides in PDF form. For more information please contact sales@solver.com.
Frontline Systems Inc. (https://www.solver.com) is the alternative to analytics complexity, helping business analysts and managers gain insights and make better decisions for an uncertain future, without the cost, delays and risk of ‘big vendor’ tools. Its products integrate forecasting and data mining for “predictive analytics,” Monte Carlo simulation for risk analysis, conventional and stochastic optimization for “prescriptive analytics,” and business rules and Excel calculations to make the best business decisions. Founded in 1987, Frontline is based in Incline Village, Nevada (775-831-0300).
Microsoft Excel and Office 365 are trademarks of Microsoft Corp. Analytic Solver®, RASON® and Solver SDK® are registered trademarks of Frontline Systems Inc.