Abstract
Test case prioritization techniques organize test cases for implementation in a manner that enhance their efficacy in accordance with some performance goal. The main aim of regression testing is to test the amended software to assure that the amendments performed in software are correct. It is not always feasible to retest entire test cases in a test suite due to limited resources. Therefore, it is necessary to develop some effective techniques that can enhance the regression testing effectiveness by organizing the test cases in an order following some testing criterion. One possible criterion of such prioritization is to enhance a test suite’s fault detection rate. It aspires to arrange test cases in an order that higher priority test cases run earlier than lower ones. This paper proposed a methodology for prioritizing regression test cases based on four factors namely the rate of fault detection, the number of faults detected, the test case ability of risk detection and the test case effectiveness. The proposed approach is implemented on two projects. The resultant test case order is analyzed with other prioritization techniques such as no prioritization, random prioritization, reverse prioritization, optimal prioritization and along with previous works for project 1. We have applied our proposed approach for prioritizing test cases in an order that maximize fault coverage with least test suite execution and compared its effectiveness with other orderings. The result of proposed approach shows higher average percentage of fault detected value and outperforms all other approaches.
Similar content being viewed by others
Change history
20 April 2017
An erratum to this article has been published.
References
Myers, G.J.: The Art of Software Testing. Wiley, New York (1979)
Pressman, R.S.: Software Engineering: A Practitioner’s Approach, 5th edn. McGraw-Hill International edition, New York (2001)
I.S.G. of Software Engg. Terminology. IEEE standards collection, IEEE std 610.12-1990 (1990)
Rothermel, G.; Untch, R.; Chu, C.; Harrold, M.: Test case prioritization: An empirical study. In: Proceedings of IEEE international conference on Software Maintenance (ICSM’ 1999), pp 179–188 (1999)
Elbaum, S.; Malishevsky, A.; Rothermel, G.: Prioritizing test cases for regression testing. In: Proceedings of the 2000 ACM SIGSOFT International Symposium on Software Testing and Analysis, Portland, Oregon, USA, pp 102–112 (2000)
Marre, M.; Bertolino, A.: Using spanning sets for coverage testing. IEEE Trans. Softw. Eng. 29(11), 974–984 (2003)
Wong, W.E.; Horgan, J.R.; London, S.; Mathur, A.P.: Effect of test set minimization on fault detection effectiveness. Softw. Pract. Exp. 28(4), 347–369 (1998)
Wong, W.E.; Horgan, J.R.; Mathur, A.P.; Pasquini, A.: Test set size minimization and fault detection effectiveness: A case study in a space application. In: Proceedings of 21st Annual International Computer Software Application Conference, pp. 522–528 (1997)
Rotherrmel, G.; Harrold, M.J.; Ostrin, J.; Hong, C.: An emprical study of the effects of minimization on the fault detection capabilities of test suites. In: Proceedings of the International Conference on Software Maintenance, pp. 34–43 (1998)
Ball, T.: On the limit of control flow analysis for regression test selection. In: Proceedings of ACM International Symposium on Software Testing and Analysis, pp. 134–142 (1998)
Chen, Y.F.; Rosenblum, D.S.; Vo, K.P.: TestTube: A system for selective regression testing. In: Proceedings of 16th International Conference of Software Engineering, pp. 211–222 (1994)
Rothermel, G.; Harrold, M.J.: A safe, efficient regression test selection technique. ACM Trans. Softw. Eng. Methodol. 6(2), 173–210 (1997)
Vokolos, F.I.; Frankl, P.G.: Pythia: A regression test selection tool based on textual differencing. In: Proceedings of Third International Conference of Reliability, Quality and Safety of Software Intensive Systems (ENCRESS 1997) (1997)
Rothermel, G.; Harrold, M.J.: Analyzing regression test selection techniques. IEEE Trans. Softw. Eng. 22(8), 529–551 (1996)
Kavitha, R.; Sureshkumar, N.: Test case prioritization for regression testing based on severity of fault. Int. J. Comput. Sci. Eng. (IJCSE) 2(5), 1462–1466 (2010)
Chernak, Y.: Validating and improving test case effectiveness. IEEE Softw. 16(1), 81–86 (2001)
Tyagi, M.; Malhotra, S.: An approach for test case prioritization based on three factors. Int. J. Inf. Technol. Comput. Sci. 4, 79–86 (2015)
Rothermel, G.; Untch, R.; Chu, C.; Harrold, M.: Prioritizing test cases for regression testing. IEEE Trans. Softw. Eng. 27(10), 929–948 (2001)
Do, H.; Elbaum, S.G.; Rothermel, G.: Supporting controlled experimentation with testing techniques: an infrastructure and its potential impact. Empir. Softw. Eng. 10(4), 405–435 (2005)
Beizer, B.: Software System Testing and Quality Assurance. Van Nostrand, New York (1984)
Elbaum, S.; Malishevsky, A.G.; Rothermel, G.: Test case prioritization: a family of empirical studies. IEEE Trans. Softw. Eng. 28(2), 159–182 (2002)
Hamlet, R.G.: Probable correctness theory. Inf. Process. Lett. 25, 17–25 (1987)
Hamlet, R.G.: Testing programs with the aid of a compiler. IEEE Trans. Softw. Eng. 3(4), 279–290 (1997)
Avritzer, A.; Weyuker, E.J.: The automatic generation of load test suites and the assessment of the resulting software. IEEE Trans. Softw. Eng. 21(9), 705–716 (1995)
Malishevsky, A.; Ruthruff, J.R.; Rothermel, G.; Elbaum, S.: Cost-cognizant test case prioritization. Technical report TR-UNL-CSE-2006-004, Dept. of CSE, University of Nebraska-Lincoln, USA (2006)
Dickinson, W.; Leon, d.; Podgurski, A.: Finding failures by cluster analysis of execution profiles. In: International Conference on Software Engineering, Toronto, Canada, pp. 339–348 (2001)
Wong, W.E.; Horgan, J.R.; London, S.; Agrawal, H.: A study of effective regression testing in practice. In: Proceedings of Eighth International Symposium on Software Reliability Engineering, pp. 230–238 (1997)
Harrold, M.J.; Gupta, R.; Soffa, M.L.: A methodology for controlling the size of a test suite. ACM Trans. Softw. Eng. Methodol. 2(3), 270–285 (1993)
Jeffrey, D.; Gupta, R.: Test suite reduction with selective redundancy. In: 21st IEEE Conference on Software Maintenance (ICSM’05), pp. 549–558 (2005)
Ren, X.; Saha, F.; Tip, F.; Ryder, B.G.; Chesley, O.: Chianti: A Tool for Change Impact Analysis of Java Programs. OOPSLA’04, Vancouver, Canada, ACM Sigplan, pp. 432-448 (2004)
Elbaum, S.; Rothermel, G.; Kanduri, S.: Selecting cost-effective test case prioritization technique. Softw. Qual. J. 12(3), 185–210 (2004)
Kim, J.; Porter, A.: A history based test prioritization technique for regression testing in resource constraint environments. In: ICSE’02, ACM, Orlando, U.S.A., pp. 119–129 (2002)
Walcott, K.R.; Soffa, M.L.; Kapfhammer, G.M.; Roos, R.S.: Time-aware test suite prioritization. In: The Proceedings of the ACM SIGSOFT/SIGPLAN International Symposium on Software Testing and Analysis, Portland, Maine (2006)
Jeffrey, D.; Gupta, R.: Test case prioritization using relevant slices. In: COMPSAC’06, IEEE International Conference, Chicago, pp. 411–420 (2006)
Srikanth, H.; Williams, L.; Osborne, J.: Towards the prioritization of system test cases. North Carolina State University TR-2005-44 (2005)
Aggarwal, K.K.; Singh, Y.; Kaur, A.; Malhotra, R.: Software reuse metrics for object-oriented systems. In: Proceedings of the Third ACIS Interanational on Conference On Software Engineering Research, Management and Applications (SERA ‘05), pp. 48–55 (2005)
Li, Z.; Harman, M.; Hierons, R.M.: Search algorithms for regression test case prioritization. IEEE Trans. Softw. Eng. 33(4), 225–237 (2007)
Masri, W.; Podgurski, A.; Leon, D.: An empirical study of test case filtering techniques based on exercising information flows. IEEE Trans. Softw. Eng. 33(7), 454–477 (2007)
Qu, B.; Nie, C.; Xu, B.; Zhang, X.: Test case prioritization for black box testing. In: Proceedings of Computer Software and Applications Conference, pp. 465–474 (2007)
Korel, B.; Koutsogiannakis, G.; Tahat, L.H.: Model-based test prioritization heuristic methods and their evaluation. In: Proceeding A-MOST’07 Proceedings of the 3rd International Workshop on Advances in Model-based Testing, ACM, NY, USA, pp. 34–43 (2007)
Kavitha, R.; Kavitha, V.R.; Suresh, N.: Requirement Based Test Case Prioritization. In: IEEE ICCCCT (2010)
Maheswari, R.; Mala, D.: A novel approach for test case prioritization. In: IEEE International Conference on Computational Intelligence and Computing Research (2013)
Kayes, I.: Test case prioritization for regression testing based on fault dependency. In: IEEE (2011)
Singh, Y.; Kaur, A.; Suri, B.: Test case prioritization using ant colony optimization. ACM SIGSOFT Softw. Eng. Notes 35(4), 1–7 (2010)
Hla, K.H.S.; Choi, Y.; Park, J.S.: Applying particle swarm optimization to prioritizing test cases for embedded real time software retesting. In: Proceedings of the IEEE 8th International Conference on Computer and Information Technology Workshops, pp. 527–532 (2008)
Sabharwal, S.; Sibal, R.C.; Sharma, C.: A genetic algorithm based approach for prioritization of test case scenarios in static testing. In: Proceedings of the 2nd International Conference on computer and Communication Technology, IEEE Xplore Press, Allahabad, pp. 304–309 (2011)
Ray, M.; Mohapatra, D.P.: Multi-Objective Test Prioritization via a Genetic Algorithm. Innovations in System Software Engineering. Springer, New York (2014)
Mala, D.J.; Kamalapriya, M.; Shobana, R.; Mohan, V.: A non-pheromone based intelligent swarm optimization technique in software test suite optimization. IEEE (2009)
Huang, Y.C.; Huang, C.Y.; Chang, J.R.; Chen, T.Y.: Design and analysis of cost cognizant test case prioritization using genetic algorithm with test history. In: 34th Annual Computer Software and Applications Conference, IEEE (2010)
Souza, L.S.; Prudencio, R.B.C.; Barros, F.D.A.: A constrained particle swarm optimization approach for test case selection. In: Proceedings of the 22nd International Conference on Software Engineering and Knowledge Engineering (SEKE 2010), Redwood City, CA, USA (2010)
Sherriff, M.; Lake, M.; Williams, L.: Prioritization of regression tests using singular value decomposition with empirical change records. In: 18th IEEE International Symposium on Software Reliability Engineering (2007)
Alsmadi, I.; Alda, S.: Test case reduction and selection optimization in testing web services. Int. J. Inf. Eng. Electron. Bus. MECS Publisher (2012)
Muthusamy, T.; Seetharaman, K.: A test case prioritization method with weight factors in regression testing based on measurement metrics. Int. J. Adv. Res. Comput. Sci. Softw. Eng. 3(12) (2013)
Kumar, S.; Singh, S.: Test case prioritization: Various techniques—A review. Int. J. Sci. Eng. Res. 4(4) (2013)
Jenifer, I.; Ilavarasi, A.K.: A review on test case prioritization techniques. IJARCSSE, 4(3) (2014)
Elbaum, S.; Malishevsky, A.; Rothermel, G.: Incorporating varying test costs and fault severities into test case prioritization. In: ICSE ’01: Proceedings of the 23rd International Conference on Software Engineering. IEEE Computer Society, Washington, pp 329–338 (2001)
Author information
Authors and Affiliations
Corresponding author
Additional information
The original version of this article was revised: At the end of Sect. 4 the algorithm after the sentence “The proposed TCP technique is presented in the form of an algorithm as follows:” had been missing.
Rights and permissions
About this article
Cite this article
Nayak, S., Kumar, C. & Tripathi, S. Enhancing Efficiency of the Test Case Prioritization Technique by Improving the Rate of Fault Detection. Arab J Sci Eng 42, 3307–3323 (2017). https://doi.org/10.1007/s13369-017-2466-6
Received:
Accepted:
Published:
Issue Date:
DOI: https://doi.org/10.1007/s13369-017-2466-6