TY - JOUR
T1 - ChatGPT vs SBST: A
Comparative Assessment of Unit Test Suite Generation
AU - Tang, Yutian
AU - Liu, Zhijie
AU - Zhou, Zhichao
AU - Luo, Xiapu
PY - 2024/6
Y1 - 2024/6
N2 - Recent advancements in large language models (LLMs) have demonstrated exceptional success in a wide range of general domain tasks, such as question answering and following instructions. Moreover, LLMs have shown potential in various software engineering applications. In this study, we present a systematic comparison of test suites generated by the ChatGPT LLM and the state-of-the-art SBST tool EvoSuite. Our comparison is based on several critical factors, including correctness, readability, code coverage, and bug detection capability. By highlighting the strengths and weaknesses of LLMs (specifically ChatGPT) in generating unit test cases compared to EvoSuite, this work provides valuable insights into the performance of LLMs in solving software engineering problems. Overall, our findings underscore the potential of LLMs in software engineering and pave the way for further research in this area.
AB - Recent advancements in large language models (LLMs) have demonstrated exceptional success in a wide range of general domain tasks, such as question answering and following instructions. Moreover, LLMs have shown potential in various software engineering applications. In this study, we present a systematic comparison of test suites generated by the ChatGPT LLM and the state-of-the-art SBST tool EvoSuite. Our comparison is based on several critical factors, including correctness, readability, code coverage, and bug detection capability. By highlighting the strengths and weaknesses of LLMs (specifically ChatGPT) in generating unit test cases compared to EvoSuite, this work provides valuable insights into the performance of LLMs in solving software engineering problems. Overall, our findings underscore the potential of LLMs in software engineering and pave the way for further research in this area.
UR - https://doi.ieeecomputersociety.org/10.1109/TSE.2024.3382365
M3 - Journal article
SN - 0098-5589
VL - 50
SP - 1340
EP - 1359
JO - IEEE Transactions on Software Engineering
JF - IEEE Transactions on Software Engineering
ER -