SpringerLink
Log in

Fault coverage-based test suite optimization method for regression testing: learning from mistakes-based approach

  • Hybrid Artificial Intelligence and Machine Learning Technologies
  • Published:
Neural Computing and Applications Aims and scope Submit manuscript

Abstract

This paper presents a novel method referred as fault coverage-based test suite optimization (FCBTSO) for regression test suite optimization. FCBTSO is proposed based on Harrolds–Gupta–Soffa (HGS) test suite reduction method, and it follows the phenomenon: “learning from mistakes”. We conducted computational experiments on 12 versions of benchmarked programs retrieved from software artefact infrastructure repository and dummy fault matrix test. The performance of the proposed FCBTSO is measured against the traditional test suite reduction methods (Greedy method, Additional Greedy, HGS, and Enhanced HGS) by following the performance measures: fault coverage, execution time and reduced optimized test suite size. Rigorous statistical tests are conducted to determine the performance significance, which indicates that FCBTSO outperforms other approaches implemented with respect to the execution time that includes the execution time of the proposed approach to find the optimized test suite and the execution time of test cases in the optimized test suite.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Spain)

Instant access to the full article PDF.

Institutional subscriptions

Fig. 1
Fig. 2
Fig. 3
Fig. 4

Similar content being viewed by others

References

  1. Eghbali S, Tahvildari L (2016) Test case prioritization using lexicographical ordering. IEEE Trans Softw Eng 42:1178–1195

    Article  Google Scholar 

  2. Augustsson A (2012) A framework for evaluating regression test selection techniques in industry. Chalmers University of Technology Gothenburg, Sweden

    Google Scholar 

  3. Yoo S, Harman M (2010) Using hybrid algorithm for pareto effcient multi-objective test suite minimisation. J Syst Softw 83:689–701

    Article  Google Scholar 

  4. Harrold MJ, Rosenblum D, Rothermel G, Weyuker E (2001) Empirical studies of a prediction model for regression test selection. IEEE Trans Softw Eng 27:248–263

    Article  Google Scholar 

  5. Malishevsky AG, Ruthruff JR, Rothermel G, Elbaum S (2006) Cost-cognizant test case prioritization

  6. Elbaum S, Malishevsky AG, Rothermel G (2002) Test case prioritization: a family of empirical studies. IEEE Trans Softw Eng 28:159–182

    Article  Google Scholar 

  7. Shi A, Gyori A, Gligoric M et al (2014) Balancing trade-offs in test-suite reduction. In: FSE’14, pp 1–11

  8. Park S, Hossain BMM, Hussain I et al (2012) CarFast: achieving higher statement coverage faster. In: Proceedings of the ACM SIGSOFT 20th international symposium on the foundations of software engineering SIGSOFT’12/FSE-20, pp 1–11

  9. Li Z, Harman M, Hierons RM (2007) Search algorithms for regression test case prioritization. IEEE Trans Softw Eng 33:225–237

    Article  Google Scholar 

  10. Wong WE, Horgan JR, London S, Agrawal H (1997) A study of effective regression testing in practice. In: Proceedings of the eighth international symposium on software reliability engineering, pp 264–274

  11. Harrold MJ, Gupta R, Soffa ML (1993) A methodology for controlling the size of a test suite. ACM Trans Softw Eng Methodol 2:270–285

    Article  Google Scholar 

  12. Gladston A, Nehemiah HK, Narayanasamy P, Kannan A (2015) Test suite reduction using HGS based heuristic approach. Comput Inform 34:1113–1132

    Google Scholar 

  13. Rothermel G, Harrold MJ (1997) A safe, efficient regression test selection technique. ACM Trans Softw Eng Methodol 6:173–210

    Article  Google Scholar 

  14. Offutt AJ, Pan J, Voas JM (1995) Procedures for reducing the size of coverage-based test sets. In: International conference on testing computer software, pp 1–11

  15. Binkley D (1997) Semantics guided regression test cost reduction. IEEE Trans Softw Eng 23:498–516

    Article  Google Scholar 

  16. Rothermel G, Harrold MJ, Von Ronne J, Hong C (2002) Empirical studies of test-suite reduction. J Softw Testing Verif Reliab 12:219–249

    Article  Google Scholar 

  17. Briand LC, Labiche Y, He S (2009) Automating regression test selection based on UML designs. Inf Softw Technol 51:16–30

    Article  Google Scholar 

  18. King JC (1975) A new approach to program testing. In: Proceedings of the international conference on reliable systems, pp 228–233

  19. Miller W, Spooner DL (1976) Automatic generation of floating-point test data. IEEE Trans Softw Eng SE 2:223–226

    Article  MathSciNet  Google Scholar 

  20. Harrold MJ, Souffa ML (1988) An incremental approach to unit testing during maintenance. In: Proceedings of the conference on software maintenance, pp 362–367

  21. Leung HKN (1989) Insights into regression testing. In: Proceedings of the international conference on software maintenance, pp 60–69

  22. Fujiwara S, Bochmann GV, Khendek F et al (1991) Test selection based on finite state models. IEEE Trans Softw Eng 17:591–603

    Article  Google Scholar 

  23. Rothermel G, Harrold MJ (1996) Analyzing regression test selection techniques. IEEE Trans Softw Eng 22:529–551

    Article  Google Scholar 

  24. Rothermel G, Untch RH, Chu C, Harrold MJ (1999) Test case prioritization: an empirical study. In: Proceedings IEEE international conference on software maintenance—1999 (ICSM’99). “Software Maintenance for Business Change”, pp 179–188

  25. Graves TL, Harrold MJ, Kim JM et al (2001) An empirical study of regression test selection techniques. ACM Trans Softw Eng Methodol 10:184–208. https://doi.org/10.1145/367008.367020

    Article  MATH  Google Scholar 

  26. Jeffrey D, Gupta N (2005) Test suite reduction with selective redundancy. In: IEEE international conference on software maintenance, ICSM, pp 549–558

  27. Grindal M, Lindström B, Offutt J, Andler SF (2006) An evaluation of combination strategies for test case selection. Empir Softw Eng 11:583–611. https://doi.org/10.1007/s10664-006-9024-2

    Article  Google Scholar 

  28. Jeffrey D, Gupta N (2007) Improving fault detection capability by selectively retaining test cases during test suite reduction. IEEE Trans Softw Eng 33:108–123

    Article  Google Scholar 

  29. Zhong H, Zhang L, Mei H (2008) An experimental study of four typical test suite reduction techniques. Inf Softw Technol 50:534–546

    Article  Google Scholar 

  30. Chen TY, Lau MF (1998) A new heuristic for test suite reduction. Inf Softw Technol 40:347–354

    Article  Google Scholar 

  31. Mansour N, El-Fakih K (1999) Simulated annealing and genetic algorithms for optimal regression testing. J Softw Maint Res Pract 11:19–34

    Article  Google Scholar 

  32. Black J, Melachrinoudis E, Kaeli D (2004) Bi-criteria models for all-uses test suite reduction. In: Proceedings of the 26th international conference on software engineering, pp 106–115

  33. Lin JW, Huang CY (2009) Analysis of test suite reduction with enhanced tie-breaking techniques. Inf Softw Technol 51:679–690

    Article  Google Scholar 

  34. Chen Y, Probert RL, Ural H (2009) Regression test suite reduction based on SDL models of system requirements. J Softw Maint Evol Res Pract 21:379–405

    Article  Google Scholar 

  35. Wang YH, Wu IC (2009) Achieving high and consistent rendering performance of java AWT/Swing on multiple platforms. Softw Pract Exp 39:701–736

    Google Scholar 

  36. Mirarab S, Akhlaghi S, Tahvildari L (2012) Size-constrained regression test case selection using multicriteria optimization. IEEE Trans Softw Eng 38:936–956

    Article  Google Scholar 

  37. Wang S, Ali S, Gotlieb A (2013) Minimizing test suites in software product lines using weight-based genetic algorithms. In: 2013 15th genetic and evolutionary computation conference, GECCO 2013, pp 1493–1500

  38. Liu P (2014) An efficient reduction approach to test suite. In: 15th IEEE/ACIS international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (SNPD), pp 1–5

  39. Harris P, Raju N (2015) A greedy approach for coverage-based test suite reduction. Int Arab J Inf Technol 12:17–23

    Google Scholar 

  40. Shi A, Yung T, Gyori A, Marinov D (2015) Comparing and combining test-suite reduction and regression test selection. In: Proceedings of the 2015 10th joint meeting on foundations of software engineering, pp 237–247

  41. Harris P, Raju N (2015) Towards test suite reduction using maximal frequent data mining concept. Int J Comput Appl Technol 52:48–58

    Article  Google Scholar 

  42. Panichella A, Oliveto R, Di Penta M, De Lucia A (2015) Improving multi-objective test case selection by injecting diversity in genetic algorithms. IEEE Trans Softw Eng 41:358–383

    Article  Google Scholar 

  43. Hafez S, Elnainay M, Abougabal M, Elshehaby S (2016) Potential-fault cache-based regression test selection. In: Proceedings of IEEE/ACS international conference on computer systems and applications, AICCSA, pp 1–8

  44. Panda S, Mohapatra DP (2017) Regression test suite minimization using integer linear programming model. Softw Pract Exp 47:1539–1560

    Article  Google Scholar 

  45. Garousi V, Özkan R, Betin-Can A (2018) Multi-objective regression test selection in practice: an empirical study in the defense software industry. Inf Softw Technol 103:40–54

    Article  Google Scholar 

  46. Chen J, Zhu L, Chen TY et al (2018) Test case prioritization for object-oriented software: an adaptive random sequence approach based on clustering. J Syst Softw 135:107–125

    Article  Google Scholar 

  47. Do H, Elbaum S, Rothermel G (2005) Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir Softw Eng 10:405–435

    Article  Google Scholar 

Download references

Author information

Authors and Affiliations

  1. Amity School of Engineering and Technology, Amity University Uttar Pradesh, Noida, India

    Arun Prakash Agrawal & Ankur Choudhary

  2. USIC&T, Guru Gobind Singh Indraprastha University, New Delhi, India

    Arvinder Kaur

  3. Edge Hill University, Ormskirk, UK

    Hari Mohan Pandey

Authors
  1. Arun Prakash Agrawal
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Ankur Choudhary
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Arvinder Kaur
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Hari Mohan Pandey
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Arun Prakash Agrawal.

Ethics declarations

Conflict of interest

The authors declare that they have no conflict of interest.

Additional information

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Agrawal, A.P., Choudhary, A., Kaur, A. et al. Fault coverage-based test suite optimization method for regression testing: learning from mistakes-based approach. Neural Comput & Applic 32, 7769–7784 (2020). https://doi.org/10.1007/s00521-019-04098-9

Download citation

  • Received:

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s00521-019-04098-9

Keywords

Search

Navigation