SpringerLink
Log in

Asymptotics of Predictive Stochastic Complexity

  • Conference paper
New Directions in Time Series Analysis

Part of the book series: The IMA Volumes in Mathematics and its Applications ((IMA,volume 46))

Abstract

The purpose of this expository paper is to give a survey of rigorous asymptotic results in the theory of stochastic complexity. We provide a brief survey of the early developments and the ideas which culminated in a basic inequality on universal codes. The rest of the paper is devoted to results showing that the lower bound given by the basic inequality is achieved in many important cases. The asymptotic behaviour of three kinds of predictive stochastic complexities associated with ARMA processes will be established. Finally, extensions of the ARMA results for multivariable systems, for Ljung’s scheme, and for nonparametric problems will be considered.

On leave from Computer and Automation Institute of the Hungarian Academy of Sciences, Budapest. The work of the first author was supported by NSERC, Canada.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
EUR 29.95
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
EUR 85.59
Price includes VAT (France)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
EUR 105.49
Price includes VAT (France)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Preview

Unable to display preview. Download preview PDF.

Unable to display preview. Download preview PDF.

Similar content being viewed by others

References

  1. H. Akaike, Statistical predictor identification Ann. Inst. Statis. Math., 22 (1970), pp.202–217.

    Google Scholar 

  2. H. Akaike, A new look at the statistical model identification IEEE Trans. Aut. Cont., AC-19 (1974), pp.716–723.

    Article  MathSciNet  Google Scholar 

  3. K.J. Åström, and B. Wittenmark, On self-tuning regulators Automatica, 9 (1973), pp.185–199.

    Article  MATH  Google Scholar 

  4. J. Baikovicius and L. Gerencsér, Change point detection in a stochastic complexity framework To appear in the Proc of the 29-th IEEE CDC (1990).

    Google Scholar 

  5. A.R. Barron and T.M. Cover, Minimum complexity density estimation Submitted to IEEE Trans. Inf. Th. (1989).

    Google Scholar 

  6. A.R. Barron and C. Sheu, Approximation of densities by sequences of exponential families Submitted to the Annals of Statistics (1988).

    Google Scholar 

  7. R.J. Bhansali, Effects of not knowing the order of an autoregressive process on the squared error of prediction-I J. Amer. Statist. Assoc, 76 (1981), pp.588–597.

    Article  MathSciNet  MATH  Google Scholar 

  8. P.E. Caines, Linear Stochastic Systems Wiley, 1988.

    MATH  Google Scholar 

  9. P.E. Caines and M. Gürsoy, On the L ∞ consistency of L 2 estimators Systems and Control Letters, 12 (1989), pp.71–76.

    Article  MathSciNet  MATH  Google Scholar 

  10. L.D. Davisson, Prediction error of stationary Gaussian time series of unknown covariance IEEE Trans. Inform. Theory, IT-19 (1965), pp.783–795.

    MathSciNet  Google Scholar 

  11. A.P. Dawid, Present Position and Potential Developments: Some Personal Views, Statistical Theory, The Prequential Approach J.R.S.S. A, Part 2, 147 (1984), pp.278–292.

    MathSciNet  MATH  Google Scholar 

  12. M. Deistler and B.D.O. Anderson, Linear dynamic errors-in-variable models. Some structure theory J. of Econometrics, 41 (1989), pp.39–63.

    Article  MathSciNet  MATH  Google Scholar 

  13. L. Gerencsér, On a class of mixing processes Stochastics, 26 (1989a), pp.165–191.

    MATH  Google Scholar 

  14. L. Gerencsér, Strong approximation of the recursive prediction error estimator of the parameters of an ARMA process McGill Research Centre for Intelligent Machines, TR-CIM 89–8, Submitted to Systems and Control Letters (1991).

    Google Scholar 

  15. L. Gerencsér, On Rissanen’s predictive stochastic complexity for stationary ARMA processes McGill Research Centre for Intelligent Machines. TR-CIM-89–5. Submitted to Statistical Planning and Inference (1992).

    Google Scholar 

  16. L. Gerencsér, Some new results in the theory of recursive identification Proc. of the 28-th IEEE Conference on Decision and Control, Tampa, Florida, Vol.1, (1989), pp. 242–248.

    Article  Google Scholar 

  17. L. Gerencsér, Asymptotics of Risannen’s predictive stochastic complexity: from parametric to nonparametric models Proc. of the 28-th IEEE CDC, Tampa, Florida, Vol. 2 (1989), pp.1210–1216.

    Google Scholar 

  18. L. Gerencsér, Strong approximation of the maximum likelihood estimator of the parameters of an ARMA process Proc. of the 28-th IEEE CDC, Tampa, Florida, Vol. 1 (1989), pp.630–631.

    Google Scholar 

  19. L. Gerencsér, Almost sure asymptotics of Rissanen’s predictive stochastic complexity In Realization and Modelling in System Theory. Proc. of the International Symposium on the MTNS-89, (eds. M.A. Kaashoek, J.H. van Schuppen and A.C.M. Ran), Vol. I (1990, pp. 429–436, Bikhauser, Boston.

    Chapter  Google Scholar 

  20. L. Gerencsér, On the martingale approximation of the estimation error of ARMA parameters Systems and Control Letters, 15 (1990), pp. 417–423.

    Article  MathSciNet  MATH  Google Scholar 

  21. L. GERENCSÉR, Rate of convergence of recursive estimators To appear in the SIAM J. Control and Opt. (1992).

    Google Scholar 

  22. L. Gerencsér, Stochastic complexity in identification and adaptive control Proc. of the 11-th IFAC World Congress, Tallin, USSR, Vol. II (1990).

    Google Scholar 

  23. L. Gerencsér, Closed loop parameter identifiability and adaptive control of a linear stochastic system Systems and Control Letters, 15 (1990) pp. 411–416.

    Article  MathSciNet  MATH  Google Scholar 

  24. L. Gerencsér, Strong approximation of vector-valued stochastic integrals Statistics and Probability Letters, 12 (1991), pp. 201–207.

    Article  MathSciNet  MATH  Google Scholar 

  25. L. Gerencsér, Fixed gain estimation of ARMA parameters Submitted to the IEEE Trans, on Automatic Control, (1991).

    Google Scholar 

  26. L. Gerencsér, Predictive stochastic complexity associated with fixed gain estimators Submitted to IEEE Trans. Automatic Control, (1991).

    Google Scholar 

  27. L. Gerencsér and J. Rissanen, A prediction bound for Gaussian ARMA processes Proc. of the 25-th IEEE Conference on Decision and Control, Athens, Vol. 3 (1986), pp.1487–1490.

    Google Scholar 

  28. L. Gerencsér and Zs. Vágó, Model selection in continuous time Proc. of the 30-th IEEE Conference on Decision and Control, Vol. 2 (1991), pp. 959–962.

    Article  Google Scholar 

  29. P. Hall and E.J. Hannan, On stochastic complexity and nonparametric density estimation Biometrika, 75 (1988), pp.705–719.

    Article  MathSciNet  MATH  Google Scholar 

  30. E.J. Hannan, and M. Deistler, The statistical theory of linear systems Wiley (1988).

    MATH  Google Scholar 

  31. E.J. Hannan, A.J. McDougall and D.S. Poskitt, Recursive estimation of autoregressions J. Roy, Stat. Soc. Ser B. 51 (1989), To appear.

    Google Scholar 

  32. E.J. Hannan and B.G. Quinn, The determination of the order of an autoregression J. Royal Statist. Soc. B., 41 (1979), pp.190–195.

    MathSciNet  MATH  Google Scholar 

  33. E.J. Hannan and J. Rissanen, Recursive estimation of ARMA order Biometrika 69, 1 (1982), pp.81–94.

    Article  MathSciNet  MATH  Google Scholar 

  34. E.M. Hemerley and M.H.A. Davis, Strong consistency of the PLS criterion for order determination of autoregressive processes Ann. Stat., 17 (1989), pp.941–946.

    Article  Google Scholar 

  35. A. Heunis, Rates of convergence for an adaptive filtering algorithm driven by stationary dependent data Submitted to SIAM J. Control and Opt. (1991).

    Google Scholar 

  36. P. Kabaila, On Rissanen’s lower bound on the accumulated prediction error J. of Time Series Analysis (1988).

    Google Scholar 

  37. L. Kavalieris, The estimation of the order of an autoregression using recursive residuals and cross-validation J. of Time Series analysis, 10 (1989), pp.271–281.

    Article  MathSciNet  MATH  Google Scholar 

  38. A.N. Kolmogorov and V.A. Upsenskii, Algorithms and randomness Theoria Veroyatnostey i ee Primeneniya (Theory of Probabilities and its Applications), 32–33 (1987), pp.425–455.

    Google Scholar 

  39. Yu. A. Kutoyants, Estimation of stochastic processes Publishing House of the Armenian Academy of Sciences, Yerevan (in Russian, 1980) and Heldermann, Berlin (1989).

    Google Scholar 

  40. T. Lai, Private communication.

    Google Scholar 

  41. A. Lempel AND J. Ziv, On the complexity of finite sequence IEEE Trans. Inf. Theory, IT-22 (1974), pp.75–81.

    MathSciNet  Google Scholar 

  42. L. Levin, On the notion of a random sequence Soviet Math. Doklady 14 (1973), pp. 1413–1416.

    MATH  Google Scholar 

  43. L. Ljung, Analysis of recursive stochastic algorithms IEEE Trans, on Automatic Control, (1977), AC-22, pp.551–575.

    Article  MathSciNet  Google Scholar 

  44. L. Ljung and P. Caines, Asymptotic normality of prediction error estimation for approximate system models Stochastics, 31 (1979), pp.29–46.

    MathSciNet  Google Scholar 

  45. L. Ljung and T. Söderström, Theory and practice of recursive identification MIT Press (1984).

    Google Scholar 

  46. J.M. Maciejowski, Model discrimination using an algorithmic information criterion Automatica, 15 (1979), pp.579–593.

    Article  MathSciNet  MATH  Google Scholar 

  47. P. Martin-Lov, The notion of redundancy and its use as a quantitative measure of the discrepancy between a statistical hypothesis and a set of observational data Sc. J. Statistics, 1 (1974), pp.3–18.

    Google Scholar 

  48. E. Parzen, Time series, statistics and information IMA Preprint #663, (1990).

    Google Scholar 

  49. J.R. Quinlan and R.L. Rivest, Inferring decision trees using minimum description length principle Information and Computation, 80 (1989), pp.227–248.

    Article  MathSciNet  MATH  Google Scholar 

  50. J. Rissanen, Modelling by shortest data description Automatica, 14 (1978), pp.465–471.

    Article  MATH  Google Scholar 

  51. J. Rissanen, Order estimation by accumulated prediction errors Essays in Time Series and Allied Processes (eds. J. Gani, M.B. Priestley), pp.55–61 (1984).

    Google Scholar 

  52. J. Rissanen, Universal coding, information, prediction, and estimation IEEE Trans. Inf. Theory, Vol. IT-30 (1984), pp.629–636.

    Article  MathSciNet  Google Scholar 

  53. J. Rissanen, Stochastic complexity and modeling Annals of Statistics 14 (1986), pp.1080–1100.

    Article  MathSciNet  MATH  Google Scholar 

  54. J. Rissanen, A predictive least squares principle IMA J. Math. Control Inform., 3 (1986b), pp.211–222.

    Article  MATH  Google Scholar 

  55. J. Rissanen, Stochastic complexity (with discussions) J. of the Royal Statistical Society, Series B, 49 (1987), No 3.

    Google Scholar 

  56. J. Rissanen, Stochastic complexity in statistical inquiry World Scientific Publisher (1989).

    MATH  Google Scholar 

  57. J. Rissanen, Proc. of 27th IEEE CDC, 3 (1988).

    Google Scholar 

  58. J. Rissanen, T.P. Speed and B. Yu, Density estimation by stochastic complexity IEEE Trans, on Information Theory, 38 (1992), pp. 315–323.

    Article  MATH  Google Scholar 

  59. G. Schwarz, Estimating the dimension of a model Annals of Statistics 6 (1978), pp.461–464.

    Article  MathSciNet  MATH  Google Scholar 

  60. R. Shibata, Asymptotically efficient selection of the order of the model for estimating parameters of a linear process The Ann. Stat., 8 (1980), pp.147–164.

    Article  MATH  Google Scholar 

  61. R. Shibata, Selection of the number of regression variables; A mini-max choice of generalized FPE’ Annals of Statistics, 38 (1980), pp.459–474.

    Google Scholar 

  62. V. Solo, On the convergence of AML IEEE Trans. Aut. Cont., AC-24 (1979), pp.958–962.

    Article  Google Scholar 

  63. T. Söderström and P. Stoica, System identification (1989), Prentice Hall.

    MATH  Google Scholar 

  64. Zs. Vágó and L. Gerencsér, Almost sure asymptotics of stochastic complexity for i.i.d. sequences Submitted for publication.

    Google Scholar 

  65. G. Yin, Recent progress in parallel stochastic approximations In: Topics in Stochastic Systems: Modelling, Identification and Adaptive Control (eds. L. Gerencser and P.E. Caines), Lecture Notes in Control and Information Sciences, 161, (1991), pp. 159–184.

    Chapter  Google Scholar 

  66. C.S. Wallace and D.M. Boulton, An information measure for classification Computing Journal, 11 (1968), pp.185–195.

    MATH  Google Scholar 

  67. M. Wax, Order selection for AR models by predictive least squares IEEE Trans., Acoustics, Speech, and Signal Processing, 36, (1988), pp.581–588.

    Article  MATH  Google Scholar 

  68. C.Z. Wei, On predictive least squares principles Submitted for publication, (1990).

    Google Scholar 

  69. A.K. Zhvonkin and L.A. Levin, The complexity of finite objects and the development of the concepts of information and randomness by means of the theory of algorithms Russian Mathematical Surveys, 25 (1970), pp. 83–124.

    Article  Google Scholar 

  70. J. Ziv and A. Lempel, Compression of individual sequences via variable-rate encoding IEEE Trans. Inform. Theory, IT-24 (1978), pp.530–536.

    Article  MathSciNet  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Department of Electrical Engineering, McGill University, Montréal, Québec, Canada

    László Gerencsér

  2. IBM Almaden Research Centre, San Jose, California, USA

    Jorma Rissanen

Authors
  1. László Gerencsér
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jorma Rissanen
    View author publications

    You can also search for this author in PubMed Google Scholar

Editor information

Editors and Affiliations

  1. Department of Statistics, University of California, Berkeley, CA, 94720, USA

    David Brillinger

  2. Department of Electrical Engineering, McGill University, Montreal, Quebec, H3A 2A7, Canada

    Peter Caines

  3. Department of Economics, University of Minnesota, Minneapolis, MN, 55455, USA

    John Geweke

  4. Department of Statistics, Texas A&M University, College Station, TX, 77843-3143, USA

    Emanuel Parzen

  5. Mathematics Department, University of California, San Diego, La Jolla, CA, 92093, USA

    Murray Rosenblatt

  6. Department of Mathematics, Boston University, Boston, MA, 02215, USA

    Murad S. Taqqu

Rights and permissions

Reprints and permissions

Copyright information

© 1993 Springer-Verlag New York, Inc.

About this paper

Cite this paper

Gerencsér, L., Rissanen, J. (1993). Asymptotics of Predictive Stochastic Complexity. In: Brillinger, D., Caines, P., Geweke, J., Parzen, E., Rosenblatt, M., Taqqu, M.S. (eds) New Directions in Time Series Analysis. The IMA Volumes in Mathematics and its Applications, vol 46. Springer, New York, NY. https://doi.org/10.1007/978-1-4613-9296-5_6

Download citation

  • DOI: https://doi.org/10.1007/978-1-4613-9296-5_6

  • Publisher Name: Springer, New York, NY

  • Print ISBN: 978-1-4613-9298-9

  • Online ISBN: 978-1-4613-9296-5

  • eBook Packages: Springer Book Archive

Publish with us

Policies and ethics

Search

Navigation