L2 Regularization

Describing L2 Regularization

• L2 regularization is regularization technique used for variable selection, roughly speaking
• L2 regularization achieves variable selection by slowly shrinking the OLS coefficients down to zero, but not exactly zero
• L2 regularization shrinks the OLS coefficients by applying a penalty term to the OLS objective function, and minimizing this modified cost function instead of the original OLS objective function
• The penalty term associated with L2 regularization is the square of the magnitude of OLS coefficients, which is the main difference between the L1 and L2 regularization methods
• L2 regularization is used in ridge regression

Mathematical Properties of L2 Regularization

• L2 regularization involves a circular-shaped constraint region
• The optimal point intersecting the elliptical contour plot (representing the objective function) and the constraint region does not lie on any axis where $\beta_{j}=0$, which provides L2 regularization with its incapability of true variable selection
• L2 regularization uses convex optimization, and causes the beta coefficients to almost converge to 0 but never to 0 exactly

Use-cases for L2 Regularization

• Coefficient estimation
• Close to variable selection, but not quite

References

• Differences between L1 and L2 Regularization
• Visualizing Regularization
• Regularization Lecture Notes
• Differences in Shrinking between L1 and L2