Information-criterion presets for sparse GLMs

Both L0glm.fit() and L0nnpois() accept a numeric lambda or a case-insensitive criterion name: "aic", "bic", "gic", "ebic", or "mbic". For example, lambda = "GIC" and lambda = "gic" are equivalent. Use preset.lambda() to calculate a preset's numeric value directly. L0glm.tune() accepts the same names through its tune argument, or tune = "cv" for cross-validation by held-out deviance.

Details

For a single response, the target information criterion has the form $$-2\ell(\widehat\beta) + \lambda k,$$ where \(\ell\) is the log-likelihood and \(k\) counts selected coefficients. These presets set the per-coefficient \(L_0\) penalty on the minus twice log-likelihood scale, rather than on the \(-\ell\) or mean-loss scale. All logarithms are natural. For ordinary GLMs, \(n\) is the number of rows of X; special families use the conventions in model_criteria. Here \(p\) is the original number of selectable predictors, excluding intercepts and other mandatory terms. The preset is resolved before any predictors are removed, including screening by explicit zero starts. A pool with no selectable predictors uses \(p=1\) in logarithms. Mandatory terms are unpenalized in the coefficient updates; scoring counts their parameter dimensions even when their fitted value is zero, adding a model-independent criterion term.

"aic": Akaike information criterion

\(\lambda = 2\). A prediction-oriented choice based on estimated expected Kullback–Leibler loss (Akaike, 1974). It often retains more variables and is not generally consistent for exact support recovery.

"bic": Bayesian information criterion

\(\lambda = \log n\). A conventional choice for identifying a parsimonious model when the candidate dimension is fixed or moderate (Schwarz, 1978). Its classical consistency guarantee requires a correctly specified identifiable model and regularity conditions; it need not control overfitting in very large model spaces.

"gic": generalized information criterion

\(\lambda = \log p\,\log\log n\). A selection-oriented choice for sparse problems, including high-dimensional settings. Fan and Tang (2013), Corollary 1, justify this choice under their assumptions on the design, response tails, signal strength and candidate model path. It is a consistent choice under those conditions, not a uniquely optimal penalty for every finite sample.

"ebic": linearized extended BIC

\(\lambda = \log n + 2\gamma_{\rm EBIC}\log p\), with package default \(\gamma_{\rm EBIC}=\max\{0,1-\log n/(2\log p)\}\). This accounts for large model spaces (Chen and Chen, 2008). The implementation replaces their combinatorial term \(2\gamma_{\rm EBIC}\log {p \choose k}\) by \(2\gamma_{\rm EBIC}k\log p\), giving a constant penalty per coefficient. With this default it reduces to BIC when \(p\le\sqrt n\). Supply a numeric lambda to use a different \(\gamma_{\rm EBIC}\); it is separate from the fitter's adaptive-ridge exponent gamma.

"mbic": modified BIC

\(\lambda = \log(np^2/16)\). This multiplicity-adjusted criterion is intended for sparse selection among many candidate predictors (Bogdan, Ghosh and Zak-Szatkowska, 2008). The constant used here follows Frommlet and Nuel (2016). It is distinct from the false-discovery-rate-oriented mBIC2 criterion.

GIC, SIC and HBIC

Zhu et al. (2020), in the ABESS paper, call the \(\log p\,\log\log n\) choice the special information criterion (SIC) and introduce it as a special case of HBIC (Wang, Kim and Li, 2013). The HBIC family uses \(\lambda=C_n\log p\), with a slowly diverging \(C_n\); choosing \(C_n=\log\log n\) gives the same per-coefficient penalty as Fan and Tang's GIC. For their Gaussian least-squares problem, $$\mathrm{SIC}(S)=n\log\{\mathrm{RSS}_S/(2n)\} + |S|\log p\,\log\log n.$$ This is the Gaussian profile-likelihood criterion up to an additive constant; the factor \(1/2\) inside the logarithm does not halve the complexity penalty. Use "gic" in this package; SIC and HBIC are discussed here as literature terminology, not additional argument values.

Approximation and scoring

Presets specify a target penalty; adaptive ridge does not guarantee a global minimizer of the associated information criterion. The internal ridge scaling \(\widetilde\lambda=\lambda\phi/4\), with dispersion \(\phi=1\) for Poisson and binomial models, follows the orthogonal Gaussian threshold calibration of Frommlet and Nuel (2016). Applying this calibration to general GLMs is an approximation. Criterion consistency also does not by itself establish consistency of the numerical algorithm. relaxed = TRUE refits selected coefficients with negligible penalization; L0glm.tune() searches nearby penalties and evaluates the requested criterion on the resulting fits.

Reported scores use the fitted likelihood and the package's parameter counts, including all mandatory coefficient dimensions and an extra parameter for each estimated dispersion. A known dispersion adds no parameter; dispersion does not change the selectable predictor pool \(p\). For ordinary multichannel fits, reported IC scores count nonzero coefficient entries, not active rows; the scalar presets do not establish a group-selection consistency result. Likelihood-based criteria are not defined by the standard R quasi-family likelihoods, so their reported scores can be NA. Scores are available through logLik(), AIC(), BIC(), GIC(), EBIC() and MBIC(); see model_criteria for sample-size and special-family parameter-count conventions.

References

Akaike, H. (1974). A new look at the statistical model identification. IEEE Transactions on Automatic Control, 19, 716–723. doi:10.1109/TAC.1974.1100705 .

Schwarz, G. (1978). Estimating the dimension of a model. The Annals of Statistics, 6, 461–464. doi:10.1214/aos/1176344136 .

Fan, Y. and Tang, C. Y. (2013). Tuning parameter selection in high dimensional penalized likelihood. Journal of the Royal Statistical Society: Series B, 75, 531–552. doi:10.1111/rssb.12001 .

Wang, L., Kim, Y. and Li, R. (2013). Calibrating nonconvex penalized regression in ultra-high dimension. The Annals of Statistics, 41, 2505–2536. doi:10.1214/13-AOS1159 .

Zhu, J., Wen, C., Zhu, J., Zhang, H. and Wang, X. (2020). A polynomial algorithm for best-subset selection problem. Proceedings of the National Academy of Sciences, 117, 33117–33123. doi:10.1073/pnas.2014241117 .

Chen, J. and Chen, Z. (2008). Extended Bayesian information criteria for model selection with large model spaces. Biometrika, 95, 759–771. doi:10.1093/biomet/asn034 .

Bogdan, M., Ghosh, J. K. and Zak-Szatkowska, M. (2008). Selecting explanatory variables with the modified version of the Bayesian information criterion. Quality and Reliability Engineering International, 24, 627–641. doi:10.1002/qre.936 .

Frommlet, F. and Nuel, G. (2016). An adaptive ridge procedure for L0 regularization. PLOS ONE, 11, e0148620. doi:10.1371/journal.pone.0148620 .

See also

preset.lambda(), L0glm.fit(), L0nnpois(), L0glm.tune()

Examples

set.seed(7)
X <- matrix(runif(120), nrow = 40, ncol = 3)
y <- rpois(40, 1 + 3 * X[, 1])
fit <- L0glm.fit(X, y, family = poisson(identity),
  nonnegative = TRUE, intercept = TRUE, lambda = "GIC",
  parallelize = FALSE)
fast <- L0nnpois(X, y, intercept = TRUE, lambda = "gic",
  parallelize = FALSE)
# Both presets use p = 3 selectable predictors, excluding the intercept.
c(fit$init.lambda, fast$init.lambda, log(3) * log(log(40)))