Abstract

Deep neural network (DNN) regressors are routinely trained with the squared-error loss, yet many applied targets (prices, returns, claims, clinical measurements) carry heavy-tailed noise and gross recording errors. This paper studies the robustness of DNN regression estimators from three complementary angles. Theoretically, we show that any stationary point of the empirical squared-error risk of a network with an output bias has finite-sample breakdown point 1/n under response contamination, whereas gradient-trained networks with a bounded score function ψ remain bounded under arbitrary label contamination, and become exactly invariant to the magnitude of outliers once their residuals saturate ψ. Because these properties make the classical breakdown point uninformative for practical networks, we adopt quantitative criteria: an empirical breakdown point, the sensitivity curve and Friedman–Nemenyi rank comparisons. Empirically, we train a 128-64-32 ReLU network with eight losses (squared error, Huber, adaptive Huber, least absolute deviations, log-cosh, Cauchy, Tukey bisquare and an LTS-type trimmed loss) on California Housing with 0–50% contaminated labels under three contamination mechanisms, with five independent seeds, and complement this with a Monte Carlo study of seven heavy-tailed noise laws, an ablation of common training choices, and seven classical and ensemble benchmarks. The squared-error network breaks down at the first contamination level (5%) in all five seeds, with a 50-fold increase in test error. Under symmetric Cauchy contamination all bounded-ψ losses remain stable up to 50%, but under one-sided or clustered contamination the monotone losses (Huber, log-cosh, LAD) break down at 20–40%, whereas the redescending Tukey and Cauchy losses and trimmed training do not break down at any tested level up to 40%. Tukey bisquare attains the best mean rank (2.14 of 8), significantly ahead of the Huber loss (5.38; Nemenyi CD = 1.48). Early stopping on a robustly scored validation split reduces the squared-error damage by a factor of 21–36 but does not prevent breakdown. We recommend redescending or trimmed objectives, with a robust warm start, as the default for DNN regression on data that may contain gross errors.

References

Download PDF
Back