Model Invariants and Functional Regularization
Abstract
When modeling data, we would like to know that our models are extracting facts about the data itself, and not about something arbitrary, like the order of the factors used in the modeling. Formally speaking, this means we want the model to be invariant with respect to certain transformations.
Here we look at different models and the nature of their invariants. We find that regression, MLE and Bayesian estimation all are invariant with respect to linear transformations, whereas regularized regressions have a far more limited set of invariants. As a result, regularized regressions produce results that are less about the data itself and more about how it is parameterized.
To correct this, we propose an alternative expression of regularization which we call functional regularization. Ridge regression and lasso are special cases of functional regularization, as is Bayesian estimation. But functional regularization gives a framework under which the models become invariant with respect to linear transformations. It is also more flexible, and easier to understand, thus yielding a number of advantages over ridge regression and lasso.
Bio
Dr. Harvey J. Stein was Head of the Quantitative Risk Analytics Group at Bloomberg, responsible for Bloomberg's credit risk and market risk models, but he left Bloomberg in March 2022, and hasn't yet started his next job. Dr. Stein is well known in the industry, having published and lectured on credit risk modeling, financial regulation, interest rate and FX modeling, CVA calculations, mortgage backed security valuation, COVID-19 data analysis, and other subjects. Dr. Stein is on the board of directors of the IAQF, a board member of the Rutgers University Mathematical Finance program, an adjunct professor at Columbia University, and organizer of the IAQF/Thalesians financial seminar series. He's also worked as a quant researcher on the Bloomberg for President campaign. He received his BA in mathematics from WPI in 1982 and his PhD in mathematics from UC Berkeley in 1991.
