3sharp, a Redmond based technical services company, has been commissioned by
Microsoft to undertake a competitive study of various anti-phishing
technologies. The results of that study were released just minutes ago.
The IE team comment on the study:
http://blogs.msdn.com/ie/archive/2006/09/28/774513.aspx
Before we proceed, I will say, right at the outset, that the only safe
antiphishing technology is one that *BLOCKS* access to known phishing sites.
Why? Because in its July report (released on 11 September 2006), the
Anti-Phishing Working Group reported 182 unique websites hosting password
stealing trojans, 1850 sites hosting password stealing malicious code (exploits)
and a large increase in traffic redirecting, also known as pharming:
http://www.antiphishing.org/reports/apwg_report_july_2006.pdf
In short, it is not enough to simply warn a user that a Web site is a known
phishing site yet still display the page. Just opening a phishing site in your
Web browser can be dangerous, even if you have absolutely no intention of
entering any information on a page, if that site attempts to infect your system
with a trojan, keylogger or other nasty. Please keep this in mind when deciding
which protective technology you wish to use. I cannot recommend strongly enough
that you choose a product that BLOCKS access to known phishing sites.
Ok, now to the results....
The products were tested using 100 known phishing URLs (which had to be tested
within 48 hours of collection) and 500 known good URLs.
The "winner", with the best overall performance, and a composite
accuracy score of 172 out of 200 was Internet Explorer 7 Beta 3 (V7.0.5450.33).
2nd place went to the NetCraft toolbar (V1.6.2) with IE6 with a score of 168 out
of 200.
A distant third was Google's Toolbar for Firefox with "Safe Browsing"
(V2.0) with Firefox 1.5.0.4 with a score of 106 out of 200.
The remaining products rated:
eBay's toolbar with AccountGuard (V2.3.1) with IE6 - 92 out of 200 (note, eBay
restricts itself to eBay and PayPal spoofs and will not detect any other type of
phish)
Earthlink's ScamBlocker (V3.1.5) with IE6 - 76 out of 200
GeoTrusts TrustWatch (V3b1) with IE6 - 67 out of 200
Netscape (V8.1) - 56 out of 200
McAfee SiteAdvisor (V1.5.0.0 build 3083) with IE6 - 3 out of 200
Total catch rate for known phish URLs - pay particular attention to the block
versus warn percentages
Mistakes made on known "good" URLs
Important tidbits
Although GeoTrust did very well with a 99% catch rate, it also had a very high
rate of false positives at 32.2%. Not only that, it does not block access to
known phishing sites.
When scoring results, a false block on a good site was scored as twice as bad as
a false warning. Allowing a good site had zero value.
The known phishing URLs were not taken from any feeds from known third-party
data providers or end users to the Microsoft Phishing Filter Service in IE7.
Known good URLS were pulled from a feed of randomly selected traffic-weighted
URLs provided by Microsoft and were independent of, and confirmed not to be
included in, the Microsoft Phishing Filter system (they are not in the Phishing
Filter white list).
The full report, and associated Press Release, can be found the URL below. The
report provides comprehensive information about how the products were tested,
the rules under which the tests were conducted, how and where the phishing URLs
and good URLs were sourced and how scores were calculated, and a full list of
the URLs used during testing is also included.
http://www.3sharp.com/projects/antiphishing
Podcast:
http://www.robichaux.net/blog/3sharp_releases_gone_phishing_study_of_a.php
FAQs about "Gone Phishing: Evaluating Anti-Phishing Tools for
Windows"
http://www.robichaux.net/blog/2006/09/frequently_asked_questions_about_3sharps.p
hp
Quick statistics about IE7's phishing filter
The Phishing Filter is a “real time” service that does not require a user to
download or regularly update a list of “bad” sites.
Microsoft has been adding up to 17,000 URLS a month to its Phishing Filter
service.
From February to Mid Aug 2006 the Phishing Filter helped block over 800,000
instances of people trying to access reported phishing websites using IE7 or
MSN/Windows Live Toolbar. This figure includes almost 500,000 blocks since IE7
Beta 2 was released.
IE7 users are reporting up to 4,500 potential phishing sites per week.
|