Analysis of the errors caused by the fragmentation of the Android ecosystem: an empirical study

Abstract

Software testing and debugging has always been a pervasive problem for software developers. Mobile applications are highly important to our lives and ensuring their correctness is challenging problem. Android is a popular platform for both developers and users as there are many kinds of devices that can run the operating system. However, because of the highly fragmented nature of the Android ecosystem, it is a complex task to verify that apps behave as expected. To provide more insight into this problem, we performed a study to learn quantitative information about the problems caused by fragmentation. We conducted our study by leveraging cloud-based testing services with existing and suitably developed test suites. We implemented this study by utilizing the Amazon Web Services Device Farm and Android Compatibility Test Suite to execute these tests on a large scale. As a preliminary study, we have focused on a subset of the Compatibility Test Suite test packages and have classified the discovered test failures. We present the results of our study and the fragmentation issues discovered, which we release to assist developers and device vendors in accounting for fragmentation inconsistencies. In future work, we see this study acting as a foundation for continued quantitative analysis on the fragmentation within the Android ecosystem.