Abstract
Variance regression allows for heterogeneous variance, or heteroscedasticity, by incorporating a regression model into the variance. This paper uses a variant of the expectation–maximisation algorithm to develop a new method for fitting additive variance regression models that allow for regression in both the mean and the variance. The algorithm is easily extended to allow for B-spline bases, thus allowing for the incorporation of a semi-parametric model in both the mean and variance. Although there are existing methods to fit these types of models, this new algorithm provides a reliable alternative approach that is not susceptible to numerical instability that can arise in this constrained estimation context. We utilise the developed algorithm with a series of simulation studies and analyse illustrative data. Various simulation studies show that the algorithm can recover the true model for a variety of scenarios. We also study automatic selection of model complexity based on information-based criteria, and show that the Akaike information criterion is useful for choosing the optimal number of knots in a B-spline model. An R package is available for implementing these methods.
Similar content being viewed by others
References
Aitkin M (1987) Modelling variance heterogeneity in normal regression using GLIM. J R Stat Soc: Ser C (Appl Stat) 36(3):332–339. https://doi.org/10.2307/2347792
Babu G (2011) Resampling methods for model fitting and model selection. J Biopharm Stat 21(6):1177–1186. https://doi.org/10.1080/10543406.2011.607749
Crisp A, Burridge J (1994) A note on nonregular likelihood functions in heteroscedastic regression models. Biometrika 81(3):585–587. https://doi.org/10.1093/biomet/81.3.585
De Boor C (1978) A Practical Guide to Splines. Applied mathematical sciences (Springer-Verlag New York Inc.) ; v. 27. Springer-Verlag, New York
Dempster AP, Laird NM, Rubin DB (1977) Maximum likelihood from incomplete data via the EM algorithm. J Roy Stat Soc: Ser B (Methodol) 39(1):1–38
Donnelly CA (1995) The spatial analysis of covariates in a study of environmental epidemiology. Stat Med 14(21–22):2393–2409. https://doi.org/10.1002/sim.4780142110
Donoghoe M, Marschner I (2018) logbin: An R package for relative risk regression using the log-binomial model. Journal of Statistical Software 86(9), 1–22. https://doi.org/10.18637/jss.v086.i09. https://www.jstatsoft.org/v086/i09
Donoghoe MW, Marschner IC (2016) Fast stable relative risk regression using an overparameterised EM algorithm. In: Proceedings of the 31st International Workshop on Statistical Modelling, vol. 1, pp. 93–98
Hastie T, Tibshirani R (1990) Generalized Additive Models, 1st, edition. Monographs on statistics and applied probability. Chapman and Hall, London
Hurvich CM, Simonoff JS, Tsai CL (1998) Smoothing parameter selection in nonparametric regression using an improved Akaike information criterion. Journal of the Royal Statistical Society. Series B (Statistical Methodology) 60(2), 271–293. http://www.jstor.org/stable/2985940
Ling N, Vieu P (2020) On semiparametric regression in functional data analysis. WIREs Computational Statistics p. e1538. https://doi.org/10.1002/wics.1538. https://onlinelibrary.wiley.com/doi/abs/10.1002/wics.1538
Liu C, Rubin DB (1994) The ECME algorithm: a simple extension of EM and ECM with faster monotone convergence. Biometrika 81(4):633–648. https://doi.org/10.2307/2337067
Lumley T, Kronmal R, Ma S (2006) Relative risk regression in medical research: models, contrasts, estimators, and algorithms. University of Washington Biostatistics Working Paper Series. Working Paper 293. http://biostats.bepress.com/uwbiostat/paper293/
Ma S (2014) A plug-in the number of knots selector for polynomial spline regression. J Nonparam Stat 26(3):489–507. https://doi.org/10.1080/10485252.2014.930143
Marschner IC (2014) Combinatorial EM algorithms. Stat Comput 24(6):921–940. https://doi.org/10.1007/s11222-013-9411-7
Marschner IC (2015) Relative risk regression for binary outcomes: Methods and recommendations. Australian & New Zealand Journal of Statistics 57(4):437–462 https://doi.org/10.1111/anzs.12131.https://onlinelibrary.wiley.com/doi/abs/10.1111/anzs.12131
McLachlan GJ, Krishnan T (2007) The EM algorithm and extensions. Wiley, New York. https://doi.org/10.1002/9780470191613
R Core Team: R: A Language and Environment for Statistical Computing. R Foundation for Statistical Computing, Vienna, Austria (2020). https://www.R-project.org/
Ramsay JO (1988) Monotone regression splines in action. Stat Sci 3(4):425–441. https://doi.org/10.1214/ss/1177012761
Robledo K (2018) VarReg: Semi-Parametric Variance Regression. https://CRAN.R-project.org/package=VarReg. R package version 1.0.2
Ruppert D (2002) Selecting the number of knots for penalized splines. J Comput Graph Stat 11(4):735–757
Ruppert D, Wand MP, Carroll RJ (2003) Semiparametric regression. Cambridge series in statistical and probabilistic mathematics. Cambridge University Press, New York
Sigrist MW (1994) Air monitoring by spectroscopic techniques. Chemical analysis. Wiley, New York
Smyth GK (2002) An efficient algorithm for REML in heteroscedastic regression. J Comput Graph Stat 11(4):836–847. https://doi.org/10.1198/106186002871
Varadhan R, Roland C (2008) Simple and globally convergent methods for accelerating the convergence of any EM algorithm. Scand J Stat 35(2):335–353. https://doi.org/10.1111/j.1467-9469.2007.00585.X
Venables WN, Ripley BD (2002) Modern Applied Statistics with S, fourth edition edn. Springer, New York. http://www.stats.ox.ac.uk/pub/MASS4
Verbyla AP (1993) Modelling variance heterogeneity: residual maximum likelihood and diagnostics. J Roy Stat Soc: Ser B (Methodol) 55(2):493–508
Wand M (2018) SemiPar: Semiparametic Regression. https://CRAN.R-project.org/package=SemiPar. R package version 1.0-4.2
Yang J, Benyamin B, McEvoy BP, Gordon S, Henders AK, Nyholt DR, Madden PA, Heath AC, Martin NG, Montgomery GW, Goddard ME, Visscher PM (2010) Common snps explain a large proportion of the heritability for human height. Nat Genet 42(7):565–9. https://doi.org/10.1038/ng.608
Zhou H, Alexander D, Lange K (2011) A quasi-newton acceleration for high-dimensional optimization algorithms. Stat Comput 21(2):261–273. https://doi.org/10.1007/s11222-009-9166-3
Author information
Authors and Affiliations
Corresponding author
Ethics declarations
Conflict of interest
The authors declare that they have no conflict of interest.
Additional information
Publisher's Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Appendices
Appendices
1.1 Fisher information matrix
If \({\varvec{\theta }}= (\beta _0,\beta _1,..., \beta _P, \alpha _0,\alpha _1, ..., \alpha _Q)\), with a total of W parameters (where \(W=P+Q+2\)) then the information matrix is the negative second matrix derivative of the log-likelihood function, which is a \(W\times W\) matrix.
The log-likelihood for our general model discussed in the previous section is
If we partially differentiate (10) with respect to \(\beta _0\), we get the following likelihood equation
This then follows on through each of the \(\beta _p\) parameters to give
For the likelihood equation for \(\alpha _0\), we get
and then for each of the \(\alpha _q\) parameters we have
Now, taking the second derivatives we obtain the following \((P+1) \times (P+1)\) matrix for the \({\varvec{\beta }}\) parameters. We refer to this matrix as \({\varvec{B}}=\left[ B_{ij}\right] \):
Now, the partial derivatives for the \(\alpha _q\) parameters form a \((Q+1) \times (Q+1)\) matrix \({\varvec{A}}\), where \({\varvec{A}}=[A_{ij}]\):
The partial derivatives of the combination of \(\beta _p\) and \(\alpha _q\) parameters reduce to zero when we take the expectation, and thus the expected information matrix is block diagonal:
where \({\varvec{0}}\) is a \((Q+1) \times (P+1)\) matrix of zeroes. Now, if we focus on the mean component of the expected information matrix, \({\varvec{B}}\),
and for the variance component of the expected information matrix, \({\varvec{A}}\),
Lastly, these matrices need to be inverted in order to obtain the standard errors of the respective parameters.
Rights and permissions
About this article
Cite this article
Robledo, K.P., Marschner, I.C. A new algorithm for fitting semi-parametric variance regression models. Comput Stat 36, 2313–2335 (2021). https://doi.org/10.1007/s00180-021-01067-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s00180-021-01067-6