Netalyzr: From Diagnostic to Research Tool

Some Internet service providers (ISPs) redirect their customers' Internet searches through third-party companies, according to Networking Group researchers. The behavior is one of several that researchers have tracked using data gathered by Netalyzr, an online tool that tests how open and transparent a user's connection to the Internet is. Netalyzr has been used more than 350,000 times since it went live in 2009, and this summer received recognition from both the Internet Society and the Federal Communications Commission.

While the test was designed to provide information that individual non-expert users could understand easily, tests run by Netalyzr on users' Internet connections have also provided data for researchers about broad trends in how people connect to the Internet.

Netalyzr's continuing popularity allows us to track and quickly understand emerging connectivity problems encountered by Internet users," said Networking Group researcher Christian Kreibich.

Using data aggregated from Netalyzr sessions, Kreibich, Nicholas Weaver, and Vern Paxson found that a dozen ISPs with millions of customers redirected some Web searches for terms related to popular brands like Apple and Safeway. The ISPs redirected all user traffic for sites like Bing or Yahoo! to third-party proxy servers. Normally these proxy servers would simply relay user requests. The proxies also monitored user requests in order to detect when the user searched for a specific keyword using the browser's search or address bar. If the search matched a keyword related to certain brands, the proxies would replace the normal search results with a redirection through affiliate marketing companies and ultimately to the related brand's Web sites. ISPs apparently earned commissions off the redirected searches, which bypassed the search engines for which they were intended. Researchers identified more than 160 search terms redirected this way, including "Dell," "Bloomingdales," "WSJ," and "CA," which was redirected to the Computer Associates' Web store. It is unclear whether the affiliate programs or companies holding the brands were aware of this behavior.

The behavior is an extension of another trend the researchers also tracked: users who misspell domain names in their browser bars are sometimes taken to an error page filled with ads, rather than a page supplied by the browser with a message that the server named in the request does not exist. It is believed that ISPs get a cut of the money paid to the marketing companies that generate the ads.

The finding that ISPs were redirecting not just misspelled domain names but also Web searches was reported in the New Scientist on August 4. The next day, the system was featured on popular technology news blog Slashdot.org. Subsequent demands for the Netalyzr test jumped from about 700 requests per day to about 700 per hour.

Also on August 5, the Federal Communications Commission announced that the Netalyzr team had won the FCC Open Internet Research Challenge. FCC Chairman Julius Genachowski said that Netalyzr, along with the other winners of the challenge, is "giving consumers and researchers the information they need to understand and monitor the free and open Internet."

Netalyzr is also one of four end-user tests recommended by the Internet Society that determine what problems users may encounter with IPv6, the new Internet protocol address standard that will eventually replace the protocol used by most Internet services today.

Netalyzr is primarily supported by the National Science Foundation, with additional support from Amazon.com, Google, and Comcast.