Credit and debit cards enable financial transactions at unattended "pay-at-the-pump" gas station terminals across North America. Attackers discreetly open these pumps and install skimmers, which copy sensitive card data. While EMV (“chip-and-PIN”) has made substantial inroads in traditional retailers, such systems have virtually no deployment at pay-at-the-pump terminals due to dramatically higher costs and logistical/regulatory constraints, leaving consumers vulnerable in these contexts. In an effort to improve security, station owners have deployed security indicators such as low-cost tamper-evident seals, and technologists have developed skimmer detection apps for mobile phones. Not only do these solutions put the onus on consumers to notice and react to security concerns at the pump, but the efficacy of these solutions has not been measured. In this paper, we evaluate the indicators available to consumers to detect skimmers. We perform a comprehensive teardown of all known skimmer detection apps for iOS and Android devices, and then conduct a forensic analysis of real-world gas pump skimmer hardware recovered by multiple law enforcement agencies. Finally, we analyze anti-skimmer mechanisms deployed by pump owners/operators, and augment this investigation with an analysis of skimmer reports and accompanying security measures collected by the Florida Department of Agriculture and Consumer Services over four years, making this the most comprehensive long-term study of such devices. Our results show that common gas pump security indicators are not only ineffective at empowering consumers to detect tampering, but may be providing a false sense of security. Accordingly, stronger, reliable, inexpensive measures must be developed to protect consumers and merchants from fraud.