Thursday 25 September 11:30 - 12:00, Red room.Richard Ford Florida Institute of TechnologyMarco Carvalho Florida Institute of Technology Many different client-side security products claim to provide protection against exploits from known and unknown vulnerabilities. However, to date, tests of exploit detection solutions have been lacking, and those that have been conducted are of limited utility. In this talk, we explore three different types of tests that could be used to measure the efficacy of products that claim to provide protection, and discuss the properties that these tests would actually measure. We argue that each test measures a different property of the protection provided. We further illustrate how existing tests can provide misleading information about the actual efficacy of measured products. At the simplest level, exploit detection can be measured by testing machines using known-vulnerable versions of software against exploits taken from Metasploit. This test, however, fails to distinguish between products that detect known exploits targeting known vulnerabilities and products that simply detect the presence of known exploit code. At the next level, tests could be conducted using new exploits for known vulnerabilities. These could be created by building new exploits for well-documented vulnerabilities; however, this test does not discriminate between products that rely on prior knowledge of the exploit conditions and those which do not. Finally, we propose a new test that we believe raises the bar in exploit detection testing: the measurement of new exploits targeting new vulnerabilities. We provide a methodology that allows testers to ethically and efficiently leverage new vulnerabilities, and consider how an unscrupulous vendor might attempt to 'game' this new methodology. We demonstrate how these tests can be used in a cost-effective manner, and show how to further enhance this test with minimal effort, by adding techniques that allow testers to discriminate between post-exploitation behavioural detection and actual detection of the exploit. Click here for more details about the conference.