SpringerLink
Log in

State of the CArt: evaluating covering array generators at scale

  • Competitions and Challenges
  • Regular
  • Published:
International Journal on Software Tools for Technology Transfer Aims and scope Submit manuscript

Abstract

Providing a reusable methodology for the evaluation of covering array generation utilities and apply it to a corpus of such tools, obtaining an overview of supported features and constraints, performance, output size, file formats, and practical considerations that may ease or hinder adoption.

Analysis of supported capabilities, input and output formats, and constraints, followed by an experimental evaluation of eight covering array generation tools, two of which were provided in updated versions, against a corpus of 295 publicly available benchmark models in six categories, producing arrays of strength two to six.

Capabilities, particularly constraint support, vary widely amongst competitors. Metaheuristic algorithms, commonly focused on postoptimization, tend to produce small arrays at the cost of performance. Approaches based on the In-Parameter-Order paradigm offer a good balance between speed and output size that may prove conducive to real-world adoption.

The choice of a covering array generation utility should be based on specific requirements related to the use case. Nevertheless, our evaluation identifies some candidates – CAgen, ACTS, and APPTS – which lead the field in terms of overall score. Others, such as PICT, offer unique features; however, a lack of standardization may lead to vendor lock-in.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Log in via an institution

Subscribe and save

Springer+ Basic
EUR 32.99 /Month
  • Get 10 units per month
  • Download Article/Chapter or Ebook
  • 1 Unit = 1 Article or 1 Chapter
  • Cancel anytime
Subscribe now

Buy Now

Price includes VAT (Brazil)

Instant access to the full article PDF.

Institutional subscriptions

Similar content being viewed by others

References

  1. Ahmed, B.S., Zamli, K.Z.: A variable strength interaction test suites generation strategy using particle swarm optimization. J. Syst. Softw. 84(12), 2171–2185 (2011)

    Article  Google Scholar 

  2. Beyer, D.: International competition on software testing (Test-Comp). In: Beyer, D., Huisman, M., Kordon, F., Steffen, B. (eds.) Tools and Algorithms for the Construction and Analysis of Systems, pp. 167–175. Springer, Cham (2019)

    Chapter  Google Scholar 

  3. Bombarda, A., Gargantini, A.: Parallel test generation for combinatorial models based on multivalued decision diagrams. In: 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 74–81 (2022)

    Chapter  Google Scholar 

  4. Bombarda, A., Gargantini, A.: Incremental generation of combinatorial test suites starting from existing seed tests. In: 2023 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW). IEEE, New York (2023)

    Google Scholar 

  5. Bombarda, A., Gargantini, A.: Design, implementation, and validation of a benchmark generator for combinatorial interaction testing tools (2024)

  6. Bombarda, A., Crippa, E., Gargantini, A.: An environment for benchmarking combinatorial test suite generators. In: 2021 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 48–56 (2021)

    Google Scholar 

  7. Cohen, D.M., Dalal, S.R., Fredman, M.L., Patton, G.C.: The AETG system: an approach to testing based on combinatorial design. IEEE Trans. Softw. Eng. 23(7), 437–444 (1997)

    Article  Google Scholar 

  8. Cohen, M.B., Dwyer, M.B., Shi, J.: Constructing interaction test suites for highly-configurable systems in the presence of constraints: a greedy approach. IEEE Trans. Softw. Eng. 34(5), 633–650 (2008)

    Article  Google Scholar 

  9. Colbourn, C.J., Cohen, M.B., Turban, R.: A deterministic density algorithm for pairwise interaction coverage. In: IASTED Conf. on Software Engineering, pp. 345–352 (2004)

    Google Scholar 

  10. Czerwonka, J.: Pairwise testing in the real world: Practical extensions to test-case scenarios. Microsoft Corporation, Software Testing Technical Articles (2008)

  11. Forbes, M., Lawrence, J., Lei, Y., Kacker, R.N., Kuhn, D.R.: Refining the in-parameter-order strategy for constructing covering arrays. J. Res. Natl. Inst. Stand. Technol. 113(5), 287 (2008)

    Article  Google Scholar 

  12. Fu, Y., Lei, Z., Cai, S., Lin, J., Wang, H.: WCA: A weighting local search for constrained combinatorial test optimization. Inf. Softw. Technol. 122, 106288 (2020)

    Article  Google Scholar 

  13. Gao, S.-W., Lv, J.-H., Du, B.-L., Colbourn, C.J., Ma, S.-L.: Balancing frequencies and fault detection in the in-parameter-order algorithm. J. Comput. Sci. Technol. 30(5), 957–968 (2015)

    Article  Google Scholar 

  14. Gargantini, A., Radavelli, M.: Migrating combinatorial interaction test modeling and generation to the web. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 308–317 (2018)

    Google Scholar 

  15. Gargantini, A., Vavassori, P.: Efficient combinatorial test generation based on multivalued decision diagrams. In: Hardware and Software: Verification and Testing, pp. 220–235. Springer, Berlin (2014)

    Chapter  Google Scholar 

  16. Garvin, B.J., Cohen, M.B., Dwyer, M.B.: An improved meta-heuristic search for constrained interaction testing. In: 2009 1st International Symposium on Search Based Software Engineering, pp. 13–22 (2009)

    Chapter  Google Scholar 

  17. Garvin, B.J., Cohen, M.B., Dwyer, M.B.: Evaluating improvements to a meta-heuristic search for constrained interaction testing. Empir. Softw. Eng. 16(1), 61–102 (2011)

    Article  Google Scholar 

  18. Järvisalo, M., Le Berre, D., Roussel, O., Simon, L.: The international sat solver competitions. AI Mag. 33(1), 89–92 (2012)

    Google Scholar 

  19. Kampel, L., Leithner, M., Simos, D.E.: Sliced AETG: a memory-efficient variant of the AETG covering array generation algorithm. Optim. Lett. 14(6), 1543–1556 (2020)

    Article  MathSciNet  Google Scholar 

  20. Kleine, K., Simos, D.E.: An efficient design and implementation of the in-parameter-order algorithm. Math. Comput. Sci. 12(1), 51–67 (2018)

    Article  MathSciNet  Google Scholar 

  21. Kuhn, D.R., Wallace, D.R., Gallo, A.M.: Software fault interactions and implications for software testing. IEEE Trans. Softw. Eng. 30(6), 418–421 (2004)

    Article  Google Scholar 

  22. Lei, Y., Tai, K.-C.: In-parameter-order: a test generation strategy for pairwise testing. In: Proceedings Third IEEE International High-Assurance Systems Engineering Symposium (Cat. No. 98EX231), pp. 254–261. IEEE, New York (1998)

    Chapter  Google Scholar 

  23. Leithner, M., Kleine, K., Cametrics, D.E.S.: A tool for advanced combinatorial analysis and measurement of test sets. In: 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 318–327. IEEE, New York (2018)

    Google Scholar 

  24. Leithner, M., Bombarda, A., Wagner, M., Gargantini, A., Simos, D.E.: Results of the CT Competition at the 11th International Workshop on Combinatorial Testing (2022)

    Google Scholar 

  25. Nie, C., Leung, H.: A survey of combinatorial testing. ACM Comput. Surv. 43(2), 1–29 (2011)

    Article  Google Scholar 

  26. Nurmela, K.J.: Upper bounds for covering arrays by tabu search. Discrete Appl. Math. 138(1–2), 143–152 (2004)

    Article  MathSciNet  Google Scholar 

  27. Stardom, J.: Metaheuristics and the Search for Covering and Packing Arrays. Simon Fraser University Burnaby (2001)

    Google Scholar 

  28. Takemura, K.: A C++ implementation of the IPO algorithm. In: 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 72–73 (2022)

    Chapter  Google Scholar 

  29. Torres-Jimenez, J., Izquierdo-Marquez, I.: Survey of covering arrays. In: 2013 15th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing, pp. 20–27. IEEE, New York (2013)

    Chapter  Google Scholar 

  30. Torres-Jimenez, J., Avila-George, H., Izquierdo-Marquez, I.: A two-stage algorithm for combinatorial testing. Optim. Lett. 11(3), 457–469 (2017)

    Article  MathSciNet  Google Scholar 

  31. Torres-Jimenez, J., Izquierdo-Marquez, I., Avila-George, H.: Methods to construct uniform covering arrays. IEEE Access 7, 42774–42797 (2019)

    Article  Google Scholar 

  32. Wagner, M., Kleine, K., Simos, D., Kuhn, R., Cagen, R.K.: A fast combinatorial test generation tool with support for constraints and higher-index. In: International Workshop on Combinatorial Testing (IWCT 2020), p. 3 (2020)

    Google Scholar 

  33. Wagner, M., Kampel, L., Simos, D.E.: Heuristically enhanced ipo algorithms for covering array generation. In: International Workshop on Combinatorial Algorithms, pp. 571–586. Springer, Berlin (2021)

    Chapter  Google Scholar 

  34. Wagner, M., Colbourn, C.J., Simos, D.E.: In-parameter-order strategies for covering perfect hash families. Appl. Math. Comput. 421, 126952 (2022)

    Article  MathSciNet  Google Scholar 

  35. Wagner, M., Leithner, M., Simos, D.E., Kuhn, R., Kacker, R.: Develo** multithreaded techniques and improved constraint handling for the tool CAgen. In: 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 87–93. IEEE, New York (2022)

    Chapter  Google Scholar 

  36. Wang, Y., Wu, H., Niu, X., Nie, C., Xu, J.: An adaptive penalty based parallel tabu search for constrained covering array generation. Inf. Softw. Technol. 143, 106768 (2022)

    Article  Google Scholar 

  37. Wang, Y., Wu, H., Niu, X., Nie, C., Xu, J.: A constrained covering array generator using adaptive penalty based parallel tabu search. In: 2022 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 82–86 (2022)

    Chapter  Google Scholar 

  38. Wu, H., Nie, C., Petke, J., Jia, Y., Harman, M.: A survey of constrained combinatorial testing (2019). Ar**v preprint. ar**v:1908.02480

  39. Yu, L., Lei, Y., Kacker, R.N., Kuhn, D.R.: ACTS: a combinatorial test generation tool. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation. IEEE, New York (2013)

    Google Scholar 

  40. Yu, L., Lei, Y., Nourozborazjany, M., Kacker, R.N., Kuhn, D.R.: An efficient algorithm for constraint handling in combinatorial test generation. In: 2013 IEEE Sixth International Conference on Software Testing, Verification and Validation, pp. 242–251. IEEE, New York (2013)

    Chapter  Google Scholar 

  41. Yu, L., Duan, F., Lei, Y., Kacker, R.N., Kuhn, D.R.: Constraint handling in combinatorial test generation using forbidden tuples. In: 2015 IEEE Eighth International Conference on Software Testing, Verification and Validation Workshops (ICSTW), pp. 1–9. IEEE, New York (2015)

    Google Scholar 

Download references

Acknowledgements

SBA Research (SBA-K1) is a COMET Centre within the COMET - Competence Centers for Excellent Technologies Programme and funded by BMK, BMAW, and the federal state of Vienna. COMET is managed by FFG. This work was performed partly under the financial assistance award 70NANB21H124 from U.S. Department of Commerce, National Institute of Standards and Technology. Andrea Bombarda is funded by PNRR ANTHEM (AdvaNced Technologies for Human-centrEd Medicine) – CUP: B53C22006700001 – Spoke 1 – Pilot 1.4.

Author information

Authors and Affiliations

  1. Department of Engineering, University of Bergamo, Bergamo, Italy

    Andrea Bombarda & Angelo Gargantini

  2. SBA Research, Vienna, Austria

    Manuel Leithner, Michael Wagner & Dimitris E. Simos

Authors
  1. Manuel Leithner
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Andrea Bombarda
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Michael Wagner
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Angelo Gargantini
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Dimitris E. Simos
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Manuel Leithner.

Ethics declarations

Competing Interests

Andrea Bombarda and Angelo Gargantini are co-authors of pMEDICI. Michael Wagner, Dimitris E. Simos, and Manuel Leithner are coauthors of CAgen.

Additional information

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Leithner, M., Bombarda, A., Wagner, M. et al. State of the CArt: evaluating covering array generators at scale. Int J Softw Tools Technol Transfer 26, 301–326 (2024). https://doi.org/10.1007/s10009-024-00745-2

Download citation

  • Accepted:

  • Published:

  • Issue Date:

  • DOI: https://doi.org/10.1007/s10009-024-00745-2

Keywords

Search

Navigation