Log in

Bayesian non-parametric detection heterogeneity in ecological models

  • Published:
Environmental and Ecological Statistics Aims and scope Submit manuscript

Abstract

Detection heterogeneity is inherent to ecological data, arising from factors such as varied terrain or weather conditions, inconsistent sampling effort, or heterogeneity of individuals themselves. Incorporating additional covariates into a statistical model is one approach for addressing heterogeneity, but there is no guarantee that any set of measurable covariates will adequately address the heterogeneity, and the presence of unmodelled heterogeneity has been shown to produce biases in the resulting inferences. Other approaches for addressing heterogeneity include the use of random effects, or finite mixtures of homogeneous subgroups. Here, we present a non-parametric approach for modeling detection heterogeneity for use in a Bayesian hierarchical framework. We employ a Dirichlet process mixture which allows a flexible number of population subgroups without the need to pre-specify this number of subgroups as in a finite mixture. We describe this non-parametric approach, then consider its use for modeling detection heterogeneity in two common ecological motifs: capture–recapture and occupancy modeling. For each, we consider a homogeneous model, finite mixture models, and the non-parametric approach. We compare these approaches using simulation, and observe the non-parametric approach as the most reliable method for addressing varying degrees of heterogeneity. We also present two real-data examples, and compare the inferences resulting from each modeling approach. Analyses are carried out using the nimble package for R, which provides facilities for Bayesian non-parametric models.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price excludes VAT (USA)
Tax calculation will be finalised during checkout.

Instant access to the full article PDF.

Fig. 1
Fig. 2
Fig. 3
Fig. 4
Fig. 5
Fig. 6
Fig. 7
Fig. 8
Fig. 9

Similar content being viewed by others

Data availability

All data used is available at github.com/danielturek/bnp-ecology-examples.

References

  • Ahmed A, **ng E (2008) Dynamic non-parametric mixture models and the recurrent Chinese restaurant process: with applications to evolutionary clustering. In: Proceedings of the 2008 SIAM international conference on data mining. SIAM, pp 219–230

  • Begon M, Harper J, Townsend C (2006) Ecology: from individuals to ecosystems, 4th edn. Wiley-Blackwell, New York

    Google Scholar 

  • Blackwell D, MacQueen J (1973) Ferguson distributions via Pólya urn schemes. Ann Stat 1:353–355

    Google Scholar 

  • Blei DM, Griffiths TL, Jordan MI (2010) The nested Chinese restaurant process and Bayesian nonparametric inference of topic hierarchies. J ACM 57(2):1–30

    Article  Google Scholar 

  • Brooks S, Gelman A, Jones G, Meng X-L (2011) Handbook of Markov Chain Monte Carlo. CRC Press, New York

    Book  Google Scholar 

  • Cubaynes S, Lavergne C, Marboutin E, Gimenez O (2012) Assessing individual heterogeneity using model selection criteria: how many mixture components in capture–recapture models? Methods Ecol Evol 3(3):564–573

    Article  Google Scholar 

  • Cubaynes S, Pradel R, Choquet R, Duchamp C, Gaillard J-M, Lebreton J-D, Marboutin E, Miquel C, Reboulet A-M, Poillot C, Taberlet P, Gimenez O (2010) Importance of accounting for detection heterogeneity when estimating abundance: the case of French wolves. Conserv Biol 24(2):621–626

    Article  PubMed  Google Scholar 

  • de Valpine P, Turek D, Paciorek CJ, Anderson-Bergman C, Lang DT, Bodik R (2017) Programming with models: writing statistical algorithms for general model structures with NIMBLE. J Comput Graph Stat 26(2):403–413

    Article  Google Scholar 

  • Diana A, Matechou E, Griffin J, Johnston A et al (2020) A hierarchical dependent Dirichlet process prior for modelling bird migration patterns in the UK. Ann Appl Stat 14(1):473–493

    Article  Google Scholar 

  • Dorazio RM, Mukherjee B, Zhang L, Ghosh M, Jelks HL, Jordan F (2008) Modeling unobserved sources of heterogeneity in animal abundance using a Dirichlet process prior. Biometrics 64(2):635–644

    Article  PubMed  Google Scholar 

  • Escobar MD (1994) Estimating normal means with a Dirichlet process prior. J Am Stat Assoc 89:268–277

    Article  Google Scholar 

  • Escobar MD, West M (1995) Bayesian density estimation and inference using mixtures. J Am Stat Assoc 90:577–588

    Article  Google Scholar 

  • Ferguson TS (1973) A Bayesian analysis of some nonparametric problems. Ann Stat 1:209–230

    Article  Google Scholar 

  • Ferguson TS (1974) Prior distribution on the spaces of probability measures. Ann Stat 2:615–629

    Article  Google Scholar 

  • Ford J, Patterson T, Bravington M (2015) Modelling latent individual heterogeneity in mark-recapture data with dirichlet process priors. ar**v:1511.07103

  • Gelman A, Hwang J, Vehtari A (2014) Understanding predictive information criteria for Bayesian models. Stat Comput 24(6):997–1016

    Article  Google Scholar 

  • Gimenez O, Viallefont A, Charmantier A, Pradel R, Cam E, Brown CR, Anderson MD, Brown MB, Covas R, Gaillard J (2008) The risk of flawed inference in evolutionary studies when detectability is less than one. Am Nat 172(3):441–448

    Article  PubMed  Google Scholar 

  • Goldstein BR, Turek D, Ponisio L, de Valpine P (2020) nimbleEcology: Distributions for ecological models in nimble. https://cran.r-project.org/package=nimbleEcology

  • Guillera-Arroita G (2017) Modelling of species distributions, range dynamics and communities under imperfect detection: advances, challenges and opportunities. Ecography 40(2):281–295

    Article  Google Scholar 

  • Jeyam A, McCrea R, Pradel R (2020) Assessing heterogeneity in transition propensity in multistate capture–recapture data. J R Stat Soc Ser C 69(2):413–427

    Article  Google Scholar 

  • Li Y, Schofield E, Gönen M (2019) A tutorial on Dirichlet process mixture modeling. J Math Psychol 91:128–144

    Article  PubMed  PubMed Central  Google Scholar 

  • Link WA (2003) Nonidentifiability of population size from capture-recapture data with heterogeneous detection probabilities. Biometrics 59(4):1123–1130

    Article  PubMed  Google Scholar 

  • Link WA (2004) Individual heterogeneity and identifiability in capture–recapture models. Anim Biodivers Conserv 27(1):87–91

    Google Scholar 

  • Lo AY (1984) On a class of Bayesian nonparametric estimates I: density estimates. Ann Stat 12:351–357

    Google Scholar 

  • Louvrier J, Chambert T, Marboutin E, Gimenez O (2018) Accounting for misidentification and heterogeneity in occupancy studies using hidden Markov models. Ecol Model 387:61–69

    Article  Google Scholar 

  • Lunn D, Spiegelhalter D, Thomas A, Best N (2009) The BUGS project: evolution, critique and future directions. Stat Med 28(25):3049–3067

    Article  PubMed  Google Scholar 

  • MacKenzie DI, Nichols JD, Royle JA, Pollock KH, Bailey L, Hines JE (2018) Occupancy estimation and modeling: inferring patterns and dynamics of species occurrence, 2nd edn. Elsevier, Amsterdam

    Google Scholar 

  • Manrique-Vallier D (2016) Bayesian population size estimation using Dirichlet process mixtures. Biometrics 72(4):1246–1254

    Article  PubMed  Google Scholar 

  • Matechou E, Caron F (2017) Modelling individual migration patterns using a Bayesian nonparametric approach for capture–recapture data. Ann Appl Stat 11(1):21–40

    Article  Google Scholar 

  • McCrea RS, Morgan B (2014) Analysis of capture–recapture data, 1st edn. Chapman & Hall, New York

    Book  Google Scholar 

  • Neal RM (2000) Markov chain sampling methods for Dirichlet process mixture models. J Comput Graph Stat 9:249–265

    Google Scholar 

  • NIMBLE Development Team (2019). Nimble: MCMC, particle filtering, and programmable hierarchical modeling. https://cran.r-project.org/package=nimble

  • Ohlssen DI, Sharples LD, Spiegelhalter DJ (2007) Flexible random-effects models using Bayesian semi-parametric models: applications to institutional comparisons. Stat Med 26(9):2088–2112

    Article  CAS  PubMed  Google Scholar 

  • Pitman J (1995) Exchangeable and partially exchangeable random partitions. Probab Theory Relat Fields 102:145–158

    Article  Google Scholar 

  • Pitman J (1996) Some developments of the blackwell-Macqueen urn scheme. In: Statistics, probability, and game theory: papers in honor of David Blackwell. JSTOR, pp 245–267

  • Pledger S (2005) The performance of mixture models in heterogeneous closed population capture–recapture. Biometrics 61(3):868–873

    Article  PubMed  Google Scholar 

  • Pledger S, Pollock KH, Norris JL (2010) Open capture-recapture models with heterogeneity: II. Jolly–Seber model. Biometrics 66(3):883–890

    Article  PubMed  Google Scholar 

  • Pohle J, Langrock R, van Beest FM, Schmidt NM (2017) Selecting the number of states in hidden Markov models: pragmatic solutions illustrated using animal movement. J Agric Biol Environ Stat 22(3):270–293

    Article  Google Scholar 

  • Qin ZS (2006) Clustering microarray gene expression data using weighted Chinese restaurant process. Bioinformatics 22(16):1988–1997

    Article  CAS  PubMed  Google Scholar 

  • Royle JA (2006) Site occupancy models with heterogeneous detection probabilities. Biometrics 62(1):97–102

    Article  PubMed  Google Scholar 

  • Sethuraman J (1994) A constructive definition of Dirichlet prior. Stat Sin 2:639–650

    Google Scholar 

  • Turek D, de Valpine P, Paciorek CJ (2016) Efficient Markov chain Monte Carlo sampling for hierarchical hidden Markov models. Environ Ecol Stat 23(4):549–564

    Article  CAS  Google Scholar 

  • Turek D, de Valpine P, Paciorek CJ, Anderson-Bergman C (2017) Automated parameter blocking for efficient Markov chain Monte Carlo sampling. Bayesian Anal 12(2):465–490

    Article  Google Scholar 

  • Warren CC, Veech JA, Weckerly FW, O’Donnell L, Ott JR (2013) Detection heterogeneity and abundance estimation in populations of golden-cheeked warblers (setophaga chrysoparia). Auk 130(4):677–688

    Article  Google Scholar 

  • Watanabe S (2010) Asymptotic equivalence of Bayes cross validation and widely applicable information criterion in singular learning theory. J Mach Learn Res 11(12):3571–3594

    Google Scholar 

  • Wehrhahn C, Rodriguez A, Paciorek C (2018) Bayesian nonparametric mixture models using nimble. In: NeurIPS workshop on nonparametric Bayesian models

Download references

Author information

Authors and Affiliations

Authors

Contributions

All authors contributed to the methodology, analysis, and manuscript preparation.

Corresponding author

Correspondence to Daniel Turek.

Ethics declarations

Funding

Support for DT was provided by Fulbright Research Scholarship Award 9183-FR and also the Williams College Class of 1945 World Fellowship. Support for CW was partially provided by award NSF-DMS 1622444. Support for OG was provided by a grant from the French Centre National de la Recherche Scientifique (CNRS) and “Mission pour l’interdisciplinarité” through its “Osezl’interdisciplinarité call.” We warmly thank the French Office of Biodiversity (OFB) for sharing the wolf datasets.

Conflict of interest

The authors have no conflicts of interest.

Code availability

Code used is available at github.com/danielturek/bnp-ecology-examples.

Additional information

Handling Editor: Luiz Duczmal

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Turek, D., Wehrhahn, C. & Gimenez, O. Bayesian non-parametric detection heterogeneity in ecological models. Environ Ecol Stat 28, 355–381 (2021). https://doi.org/10.1007/s10651-021-00489-1

Download citation

  • Received:

  • Revised:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10651-021-00489-1

Keywords

Navigation