Stateful testing: Finding more errors in code and contracts

Yi Wei; Hannes Roth; Carlo A. Furia; Yu Pei; Alexander Horton; Michael Steindorfer; Martin Nordio; Bertrand Meyer

doi:10.1109/ASE.2011.6100094

Stateful testing: Finding more errors in code and contracts

Yi Wei, Hannes Roth, Carlo A. Furia, Yu Pei, Alexander Horton, Michael Steindorfer, Martin Nordio, Bertrand Meyer

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

9 Citations (Scopus)

Abstract

Automated random testing has shown to be an effective approach to finding faults but still faces a major unsolved issue: how to generate test inputs diverse enough to find many faults and find them quickly. Stateful testing, the automated testing technique introduced in this article, generates new test cases that improve an existing test suite. The generated test cases are designed to violate the dynamically inferred contracts (invariants) characterizing the existing test suite. As a consequence, they are in a good position to detect new faults, and also to improve the accuracy of the inferred contracts by discovering those that are unsound. Experiments on 13 data structure classes totalling over 28,000 lines of code demonstrate the effectiveness of stateful testing in improving over the results of long sessions of random testing: stateful testing found 68.4% new faults and improved the accuracy of automatically inferred contracts to over 99%, with just a 7% time overhead.

Original language	English
Title of host publication	2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings
Pages	440-443
Number of pages	4
DOIs	https://doi.org/10.1109/ASE.2011.6100094
Publication status	Published - 1 Dec 2011
Externally published	Yes
Event	2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011 - Lawrence, KS, United States Duration: 6 Nov 2011 → 10 Nov 2011

Conference

Conference	2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011
Country/Territory	United States
City	Lawrence, KS
Period	6/11/11 → 10/11/11

Keywords

automation
dynamic analysis
random testing

ASJC Scopus subject areas

Software

More information

10.1109/ASE.2011.6100094

Cite this

@inproceedings{8ce5cf4a0f6c4cc4849543c4be31a2bc,

title = "Stateful testing: Finding more errors in code and contracts",

abstract = "Automated random testing has shown to be an effective approach to finding faults but still faces a major unsolved issue: how to generate test inputs diverse enough to find many faults and find them quickly. Stateful testing, the automated testing technique introduced in this article, generates new test cases that improve an existing test suite. The generated test cases are designed to violate the dynamically inferred contracts (invariants) characterizing the existing test suite. As a consequence, they are in a good position to detect new faults, and also to improve the accuracy of the inferred contracts by discovering those that are unsound. Experiments on 13 data structure classes totalling over 28,000 lines of code demonstrate the effectiveness of stateful testing in improving over the results of long sessions of random testing: stateful testing found 68.4% new faults and improved the accuracy of automatically inferred contracts to over 99%, with just a 7% time overhead.",

keywords = "automation, dynamic analysis, random testing",

author = "Yi Wei and Hannes Roth and Furia, {Carlo A.} and Yu Pei and Alexander Horton and Michael Steindorfer and Martin Nordio and Bertrand Meyer",

year = "2011",

month = dec,

day = "1",

doi = "10.1109/ASE.2011.6100094",

language = "English",

isbn = "9781457716393",

pages = "440--443",

booktitle = "2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings",

note = "2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011 ; Conference date: 06-11-2011 Through 10-11-2011",

}

Wei, Y, Roth, H, Furia, CA, Pei, Y, Horton, A, Steindorfer, M, Nordio, M & Meyer, B 2011, Stateful testing: Finding more errors in code and contracts. in 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings., 6100094, pp. 440-443, 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Lawrence, KS, United States, 6/11/11. https://doi.org/10.1109/ASE.2011.6100094

Stateful testing: Finding more errors in code and contracts. / Wei, Yi; Roth, Hannes; Furia, Carlo A. et al.
2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings. 2011. p. 440-443 6100094.

Research output: Chapter in book / Conference proceeding › Conference article published in proceeding or book › Academic research › peer-review

TY - GEN

T1 - Stateful testing: Finding more errors in code and contracts

AU - Wei, Yi

AU - Roth, Hannes

AU - Furia, Carlo A.

AU - Pei, Yu

AU - Horton, Alexander

AU - Steindorfer, Michael

AU - Nordio, Martin

AU - Meyer, Bertrand

PY - 2011/12/1

Y1 - 2011/12/1

N2 - Automated random testing has shown to be an effective approach to finding faults but still faces a major unsolved issue: how to generate test inputs diverse enough to find many faults and find them quickly. Stateful testing, the automated testing technique introduced in this article, generates new test cases that improve an existing test suite. The generated test cases are designed to violate the dynamically inferred contracts (invariants) characterizing the existing test suite. As a consequence, they are in a good position to detect new faults, and also to improve the accuracy of the inferred contracts by discovering those that are unsound. Experiments on 13 data structure classes totalling over 28,000 lines of code demonstrate the effectiveness of stateful testing in improving over the results of long sessions of random testing: stateful testing found 68.4% new faults and improved the accuracy of automatically inferred contracts to over 99%, with just a 7% time overhead.

AB - Automated random testing has shown to be an effective approach to finding faults but still faces a major unsolved issue: how to generate test inputs diverse enough to find many faults and find them quickly. Stateful testing, the automated testing technique introduced in this article, generates new test cases that improve an existing test suite. The generated test cases are designed to violate the dynamically inferred contracts (invariants) characterizing the existing test suite. As a consequence, they are in a good position to detect new faults, and also to improve the accuracy of the inferred contracts by discovering those that are unsound. Experiments on 13 data structure classes totalling over 28,000 lines of code demonstrate the effectiveness of stateful testing in improving over the results of long sessions of random testing: stateful testing found 68.4% new faults and improved the accuracy of automatically inferred contracts to over 99%, with just a 7% time overhead.

KW - automation

KW - dynamic analysis

KW - random testing

UR - http://www.scopus.com/inward/record.url?scp=84855454244&partnerID=8YFLogxK

U2 - 10.1109/ASE.2011.6100094

DO - 10.1109/ASE.2011.6100094

M3 - Conference article published in proceeding or book

SN - 9781457716393

SP - 440

EP - 443

BT - 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011, Proceedings

T2 - 2011 26th IEEE/ACM International Conference on Automated Software Engineering, ASE 2011

Y2 - 6 November 2011 through 10 November 2011

ER -

Stateful testing: Finding more errors in code and contracts

Abstract

Conference

Keywords

ASJC Scopus subject areas

More information

Other files and links

Fingerprint

Cite this