Wednesday 5 October 14:00 - 14:30, Green roomZhaoyan Xu (Palo Alto Networks)
Jun Wang (Palo Alto Networks)
Yucheng Zhou (Palo Alto Networks)
Wei Xu (Palo Alto Networks)
Kyle Sanders (Palo Alto Networks)With the technical evolution of large-scale computing and data collection, Internet-wide probing and crawling has gained more and more public acceptance in the security community. For instance, we can probe/scan network servers with delicately constructed requests to expose C&C servers. Also, crawling specific URLs can help us to identify compromised websites and Internet-wide malicious campaigns. Unlike passive monitoring and detection of ongoing attacks, the philosophy of probing and crawling aims at active and progressive searching for malicious infrastructure.However, along with the awareness of the black side, the effectiveness and efficiency of the methodology has been greatly questioned in practice. Some fundamental challenges include: how to generate a good request that detects malicious infrastructure; how to conduct a large-scale probing in a polite and effective way without disturbing normal Internet traffic; how to avoid being fingerprinted by malicious entities; how to coordinate the probing operation in a distributed fashion.Based on our two-year experience of conducting Internet-wide operations, in this paper, we discuss the challenges, methodology, system design and evaluation schemes of practical probing and crawling. In particular:Click here for more details about the conference.