IMSI catchers, also known as fake base station threats, have recently become a real concern. There are currently a few freely available tools to detect such threats, most of which are Android apps that warn users when they are connected to the fake cellular base station.
In this paper, we evaluate these Android apps and test how resistant they are against various attacking techniques. Such an evaluation is important for not only measuring the available defense against IMSI catchers attacks but also identifying gaps to build effective solutions. We developed White-Stingray, a systematic framework with various attacking capabilities in 2G and 3G networks, and used it for our study. Our results of five popular Android apps are worrisome: none of these apps are resistant to basic privacy identifier catching techniques. Based on our results, we identify limitation of these apps and propose remedies for improving the current state of IMSI catchers detection on mobile devices.