SpringerLink
Log in

Engineering Runtime Root Cause Analysis of Detected Anomalies

  • Chapter
  • First Online:
Transactions on Large-Scale Data- and Knowledge-Centered Systems LV

Part of the book series: Lecture Notes in Computer Science ((TLDKS,volume 14280))

  • 98 Accesses

Abstract

The main objective of this work is to provide a unified, easy to configure and extensible end-to-end system that performs root cause analysis (RCA) methods on top of anomaly detection (AD) methods in an online setting. AD-focused RCA for online settings has not been investigated so far; therefore our work can be seen as an initial approach to this end. Inspired by the solutions developed in the ThirdEye project, which is coupled with the Apache Pinot data warehousing system, we re-engineer ThirdEye’s RCA components/techniques so that they operate in a manner that they can directly ingest input records from Apache Kafka and continuously compute aggregates at different level of granularity in a principled manner for both OLAP queries and provision of baselines to support RCA. To attain scalability, we build our solution in the Apache Flink stream processing engine. This work presents the main design choices when applying ThirdEye’s concepts on data streams and presents indicative examples and scalability experiments. Our solution is provided in open-source.

This is a preview of subscription content, log in via an institution to check access.

Access this chapter

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Chapter
USD 29.95
Price excludes VAT (USA)
  • Available as PDF
  • Read on any device
  • Instant download
  • Own it forever
eBook
USD 49.99
Price excludes VAT (USA)
  • Available as EPUB and PDF
  • Read on any device
  • Instant download
  • Own it forever
Softcover Book
USD 64.99
Price excludes VAT (USA)
  • Compact, lightweight edition
  • Dispatched in 3 to 5 business days
  • Free ship** worldwide - see info

Tax calculation will be finalised at checkout

Purchases are for personal use only

Institutional subscriptions

Similar content being viewed by others

Notes

  1. 1.

    This is a feature deprecated in StarTree version of ThirdEye (but still available in the archived version).

  2. 2.

    It is important to note that in [41], it is stated that the original DCA is part of the enterprise edition of the StarTree ThirdEye, while for the community edition, there is a simpler algorithm that does not consider multiple dimensions jointly when making the analysis.

References

  1. Abuzaid, F., et al.: MacroBase. ACM Trans. Database Syst. 43(4), 1–45 (2018). https://doi.org/10.1145/3276463

    Article  MathSciNet  Google Scholar 

  2. Angiulli, F., Fassetti, F.: Distance-based outlier queries in data streams: the novel task and algorithms. Data Min. Knowl. Disc. 20(2), 290–324 (2010). https://doi.org/10.1007/s10618-009-0159-9

    Article  MathSciNet  Google Scholar 

  3. Apache: Apache flink: Stateful computations over data streams (2023). https://flink.apache.org/

  4. Apache: Apache kafka (2023). https://kafka.apache.org/

  5. Apache: Apache pinot: Realtime distributed olap datastore, designed to answer olap queries with low latency (2023). https://pinot.apache.org/

  6. Apache: Apache zookeeper (2023). https://zookeeper.apache.org/

  7. Apache: Flink, operators, windows (2023). https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/windows/

  8. Apache: Kafka connect overview (2023). https://kafka.apache.org/documentation/#connect

  9. Basu, S., Meckesheimer, M.: Automatic outlier detection for time series: an application to sensor data. Knowl. Inf. Syst. 11(2), 137–154 (2006). https://doi.org/10.1007/s10115-006-0026-6

    Article  Google Scholar 

  10. Blázquez-García, A., Conde, A., Mori, U., Lozano, J.A.: A review on outlier/anomaly detection in time series data. ACM Comput. Surv. 54(3), 1–33 (2021). https://doi.org/10.1145/3444690

    Article  Google Scholar 

  11. Campos, G.O., et al.: On the evaluation of unsupervised outlier detection: measures, datasets, and an empirical study. Data Min. Knowl. Discov. 30(4), 891–927 (2016)

    Article  MathSciNet  Google Scholar 

  12. Čampulová, M., Michálek, J., Mikuška, P., Bokal, D.: Nonparametric algorithm for identification of outliers in environmental data. J. Chemom. 32(5), e2997 (2018). https://doi.org/10.1002/cem.2997

    Article  Google Scholar 

  13. Carter, K.M., Streilein, W.W.: Probabilistic reasoning for streaming anomaly detection. In: 2012 IEEE Statistical Signal Processing Workshop (SSP). IEEE, August 2012. https://doi.org/10.1109/ssp.2012.6319708

  14. Chang, Y.J.: Analyzing anomalies with thirdeye (2020). https://engineering.linkedin.com/blog/2020/analyzing-anomalies-with-thirdeye

  15. Chen, J., Li, W., Lau, A., Cao, J., Wang, K.: Automated load curve data cleansing in power systems. IEEE Trans. Smart Grid 1(2), 213–221 (2010). https://doi.org/10.1109/tsg.2010.2053052

    Article  Google Scholar 

  16. Confluent: Kafka connect confluent documentation (2023). https://docs.confluent.io/platform/current/connect/index.html

  17. Docker: Docker: Accelerated, containerized application development (2023). https://www.docker.com/

  18. Flokas, Z.: Github repository: Zisisfl/online-anomaly-detection-root-cause-analysis (2023). https://github.com/ZisisFl/Online-Anomaly-Detection-Root-Cause-Analysis

  19. Flokas, Z.: Online anomaly detection and root cause analysis (msc thesis) (2023). http://ikee.lib.auth.gr/record/347173/files/GRI-2023-38956.pdf

  20. Fu, Y., Soman, C.: Real-time data infrastructure at uber. In: Proceedings of the 2021 International Conference on Management of Data. ACM, June 2021. https://doi.org/10.1145/3448016.3457552

  21. Goldstein, M., Uchida, S.: A comparative evaluation of unsupervised anomaly detection algorithms for multivariate data. PLoS ONE 11(4), e0152173 (2016)

    Article  Google Scholar 

  22. Guidotti, R., Monreale, A., Ruggieri, S., Turini, F., Giannotti, F., Pedreschi, D.: A survey of methods for explaining black box models. ACM Comput. Surv. 51(5), 1–42 (2018). https://doi.org/10.1145/3236009

    Article  Google Scholar 

  23. Gupta, N., Eswaran, D., Shah, N., Akoglu, L., Faloutsos, C.: Beyond outlier detection: LookOut for pictorial explanation. In: Berlingerio, M., Bonchi, F., Gärtner, T., Hurley, N., Ifrim, G. (eds.) ECML PKDD 2018. LNCS (LNAI), vol. 11051, pp. 122–138. Springer, Cham (2019). https://doi.org/10.1007/978-3-030-10925-7_8

    Chapter  Google Scholar 

  24. Han, J., Kamber, M., Pei, J.: Data Mining: Concepts and Techniques, 3rd edn. Morgan Kaufmann (2011). http://hanj.cs.illinois.edu/bk3/

  25. Holešovský, J., Čampulová, M., Michálek, J.: Semiparametric outlier detection in nonstationary times series: case study for atmospheric pollution in Brno, Czech republic. Atmos. Pollut. Res. 9(1), 27–36 (2018). https://doi.org/10.1016/j.apr.2017.06.005

    Article  Google Scholar 

  26. Holt, C.C.: Forecasting seasonals and trends by exponentially weighted moving averages. Int. J. Forecast. 20(1), 5–10 (2004). https://doi.org/10.1016/j.ijforecast.2003.09.015

    Article  Google Scholar 

  27. Ishimtsev, V., Nazarov, I., Bernstein, A., Burnaev, E.: Conformal K-NN anomaly detector for univariate data streams (2017)

    Google Scholar 

  28. Jacob, V., Song, F., Stiegler, A., Rad, B., Diao, Y., Tatbul, N.: Exathlon: a benchmark for explainable anomaly detection over time series. Proc. VLDB Endow. 14(11), 2613–2626 (2021)

    Article  Google Scholar 

  29. Keller, F., Muller, E., Bohm, K.: HiCS: high contrast subspaces for density-based outlier ranking. In: 2012 IEEE 28th International Conference on Data Engineering. IEEE, April 2012. https://doi.org/10.1109/icde.2012.88

  30. Keller, F., Müller, E., Wixler, A., Böhm, K.: Flexible and adaptive subspace search for outlier analysis. In: CIKM. ACM Press (2013). https://doi.org/10.1145/2505515.2505560

  31. Ma, P., Ding, R., Han, S., Zhang, D.: Metainsight: automatic discovery of structured knowledge for exploratory data analysis. In: Li, G., Li, Z., Idreos, S., Srivastava, D. (eds.) SIGMOD ’21: International Conference on Management of Data, Virtual Event, China, 20–25 June, 2021, pp. 1262–1274. ACM (2021)

    Google Scholar 

  32. Mehrang, S., Helander, E., Pavel, M., Chieh, A., Korhonen, I.: Outlier detection in weight time series of connected scales. In: 2015 IEEE International Conference on Bioinformatics and Biomedicine (BIBM). IEEE, November 2015. https://doi.org/10.1109/bibm.2015.7359896

  33. Myrtakis, N., Christophides, V., Simon, E.: A comparative evaluation of anomaly explanation algorithms (2021). https://doi.org/10.5441/002/EDBT.2021.10

    Article  Google Scholar 

  34. Ntroumpogiannis, A., Giannoulis, M., Myrtakis, N., Christophides, V., Simon, E., Tsamardinos, I.: A meta-level analysis of online anomaly detectors. VLDB J. 32(4), 845–886 (2023)

    Article  Google Scholar 

  35. Panjei, E., Gruenwald, L., Leal, E., Nguyen, C., Silvia, S.: A survey on outlier explanations. VLDB J. 31(5), 977–1008 (2022)

    Article  Google Scholar 

  36. Paparrizos, J., Kang, Y., Boniol, P., Tsay, R., Palpanas, T., Franklin, M.J.: TSB-UAD: an end-to-end benchmark suite for univariate time-series anomaly detection. Proc. VLDB Endow. 15(8), 1697–1711 (2022). https://www.vldb.org/pvldb/vol15/p1697-paparrizos.pdf

  37. Reddy, A., et al.: Using gaussian mixture models to detect outliers in seasonal univariate network traffic. In: 2017 IEEE Security and Privacy Workshops (SPW). IEEE, May 2017. https://doi.org/10.1109/spw.2017.9

  38. Ribeiro, M.T., Singh, S., Guestrin, C.: “Why should i trust you?”: explaining the predictions of any classifier (2016). https://doi.org/10.48550/ARXIV.1602.04938

  39. Schmidl, S., Wenig, P., Papenbrock, T.: Anomaly detection in time series: A comprehensive evaluation 15(9), 1779–1797. https://doi.org/10.14778/3538598.3538602

  40. Song, S., Zhang, A., Wang, J., Yu, P.S.: SCREEN. In: Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data. ACM, May 2015. https://doi.org/10.1145/2723372.2723730

  41. StarTree: Rca - top contributors (2023). https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/concepts/rca-top-contributors

  42. StarTree: Startree thirdeye (2023). https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/

  43. StarTree: Startree thirdeye community edition (2023). https://github.com/startreedata/thirdeye

  44. StarTree: Startree thirdeye product features: Community vs enterprise edition (2023). https://dev.startree.ai/docs/startree-enterprise-edition/startree-thirdeye/ThirdEyeCommEdVsEntEdition

  45. Taha, A., Hadi, A.S.: Anomaly detection methods for categorical data: a review. ACM Comput. Surv. 52(2), 38:1–38:35 (2019)

    Google Scholar 

  46. ThirdEye: Thirdeye (2019). https://thirdeye.readthedocs.io/en/latest/

  47. ThirdEye: Thirdeye archived github project (2022). https://github.com/project-thirdeye/thirdeye

  48. Toliopoulos, T., Bellas, C., Gounaris, A., Papadopoulos, A.: PROUD: PaRallel OUtlier detection for streams. In: Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data. ACM, May 2020. https://doi.org/10.1145/3318464.3384688

  49. Toliopoulos, T., Gounaris, A.: Explainable distance-based outlier detection in data streams. IEEE Access 10, 47921–47936 (2022). https://doi.org/10.1109/ACCESS.2022.3172345

    Article  MATH  Google Scholar 

  50. TPC: Tpc-ds: Decision support benchmark (2023). https://www.tpc.org/tpcds/

  51. Vinh, N.X., et al.: Discovering outlying aspects in large datasets. Data Min. Knowl. Disc. 30(6), 1520–1555 (2016). https://doi.org/10.1007/s10618-016-0453-2

    Article  MathSciNet  MATH  Google Scholar 

  52. Winters, P.R.: Forecasting sales by exponentially weighted moving averages. Manage. Sci. 6(3), 324–342 (1960). https://doi.org/10.1287/mnsc.6.3.324

    Article  MathSciNet  MATH  Google Scholar 

  53. Zhang, A., Song, S., Wang, J.: Sequential data cleaning. In: Proceedings of the 2016 International Conference on Management of Data. ACM, June 2016. https://doi.org/10.1145/2882903.2915233

  54. Zhang, H., Diao, Y., Meliou, A.: Exstream: explaining anomalies in event stream monitoring. In: International Conference on Extending Database Technology (2017)

    Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Aristotle University of Thessaloniki, Thessaloniki, Greece

    Zisis Flokas & Anastasios Gounaris

Authors
  1. Zisis Flokas
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Anastasios Gounaris
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Zisis Flokas .

Editor information

Editors and Affiliations

  1. Paul Sabatier University, Toulouse, France

    Abdelkader Hameurlain

  2. Technische Universität Wien, Wien, Austria

    A Min Tjoa

Rights and permissions

Reprints and permissions

Copyright information

© 2023 The Author(s), under exclusive license to Springer-Verlag GmbH, DE, part of Springer Nature

About this chapter

Check for updates. Verify currency and authenticity via CrossMark

Cite this chapter

Flokas, Z., Gounaris, A. (2023). Engineering Runtime Root Cause Analysis of Detected Anomalies. In: Hameurlain, A., Tjoa, A.M. (eds) Transactions on Large-Scale Data- and Knowledge-Centered Systems LV. Lecture Notes in Computer Science(), vol 14280. Springer, Berlin, Heidelberg. https://doi.org/10.1007/978-3-662-68100-8_3

Download citation

  • DOI: https://doi.org/10.1007/978-3-662-68100-8_3

  • Published:

  • Publisher Name: Springer, Berlin, Heidelberg

  • Print ISBN: 978-3-662-68099-5

  • Online ISBN: 978-3-662-68100-8

  • eBook Packages: Computer ScienceComputer Science (R0)

Publish with us

Policies and ethics

Search

Navigation