I’ve been wondering recently about how the infrastructure of the Internet really works.
I know that I have an Internet Service Provider (ISP) that supplies my connection to the Internet.
But what I don’t know is: Who provides the Internet to the ISP? And who supplies it to them? Is there a never-ending loop that eventually connects us all together?
Solution:
How do I figure out the Internet’s infrastructure?
Let’s suppose we don’t know about the history of the Internet, nor do we have access to any online resources that explain us this. Then, the only way to learn how the Internet infastructure is built is to go back to the roots. Using existing protocols to discover how our Internet is built.
Specifically, the Internet Control Message Protocol or ICMP defines the Echo request and the Echo reply. By increasing the Time To Live of IP packets by 1 each iteration, you can find each next hop on the path to your target. This allows you to get a list of hops between you and your target, the classic traceroute.
On Windows, you can use tracert; on Linux and Mac OS X, you can use traceroute.
So, let’s do a traceroute from Belgium to the United States; Stack Exchange looks like a good target.
Tracing route to stackexchange.com [64.34.119.12] over a maximum of 30 hops: ... redacted ... 5 10 ms 12 ms 12 ms te-3-3.car2.Brussels1.Level3.net [212.3.237.53] 6 11 ms 11 ms 15 ms ae-0-11.bar2.Brussels1.Level3.net [4.69.148.178] 7 20 ms 13 ms 15 ms ae-7-7.ebr1.London1.Level3.net [4.69.148.182] 8 16 ms 16 ms 18 ms vlan101.ebr2.London1.Level3.net [4.69.143.86] 9 83 ms 84 ms 87 ms ae-44-44.ebr1.NewYork1.Level3.net [4.69.137.78] 10 84 ms 93 ms 97 ms ae-71-71.csw2.NewYork1.Level3.net [4.69.134.70] 11 87 ms 96 ms 83 ms ae-2-70.edge1.NewYork1.Level3.net [4.69.155.78] 12 84 ms 93 ms 84 ms gig2-0.nyc-gsr-b.peer1.net [216.187.123.5] 13 87 ms 84 ms 85 ms gwny01.stackoverflow.com [64.34.41.58] 14 87 ms 82 ms 87 ms stackoverflow.com [64.34.119.12]Interesting, we now know that Belgium, London and New York are all connected to Level3. Level3 can be seen as an ISP to ISPs, they simply interconnect multiple ISPs. Here is a picture of how it’s connected:
Let’s go the opposite direction, China! The first thing I could find is the search engine Baidu.
Tracing route to baidu.com [123.125.114.144] over a maximum of 30 hops: ... redacted ... 5 12 ms 10 ms 12 ms ae0.anr11.ip4.tinet.net [77.67.65.177] 6 167 ms 167 ms 167 ms xe-5-1-0.sjc10.ip4.tinet.net [89.149.185.161] 7 390 ms 388 ms 388 ms as4837.ip4.tinet.net [77.67.79.150] 8 397 ms 393 ms 397 ms 219.158.30.41 9 892 ms * 392 ms 219.158.97.13 10 407 ms 403 ms 403 ms 219.158.11.197 11 452 ms 451 ms 452 ms 219.158.15.5 12 * 434 ms 434 ms 123.126.0.66 13 449 ms 450 ms 450 ms 61.148.3.34 14 432 ms 433 ms 431 ms 202.106.43.66 15 435 ms 435 ms 436 ms 123.125.114.144Well, not much information about the Chinese ISPs there but we at least found Tinet. Here is a nice picture of their site that shows how they connect with the various ISPs:
They simply have a cloud of hops spread about the relevant part of the world they serve, and at the end points they connect to the ISPs. The reason they have a cloud of hops is for reliability, for when some hops fall out…
If you repeat this a few times, you can get an idea of how everything is connected.
So, what Network Tiers are there?
The huge networks we found through trace-routing are known as Tier 1 networks.
Although there is no authority that defines tiers of networks participating in the Internet, the most common definition of a tier 1 network is one that can reach every other network on the Internet without purchasing IP transit or paying settlements.
By this definition, a tier 1 network is a transit-free network that peers with every other tier-1 network. But not all transit-free networks are tier 1 networks. It is possible to become transit-free by paying for peering or agreeing to settlements.
Common definitions of tier 2 and tier 3 networks:
Tier 2: A network that peers with some networks, but still purchases IP transit or pays settlements to reach at least some portion of the Internet.
Tier 3: A network that solely purchases transit from other networks to reach the Internet.
If you click through to Tier 1 networks from the Internet Backbone page you get to a list of the current Tier 1 networks:
- AT&T from USA
- Cogent Communications from USA
- Centurylink (formerly Qwest and Savvis) from USA
- Deutsche Telekom AG from Germany
- GTT (formerly Tinet) from USA/Italy
- Level 3 Communications from USA
- Telecom Italia Sparkle from Italy
- Telefonica Global Solutions from Spain
- Verizon Business (formerly UUNET) from USA
- TeliaSonera International Carrier from Sweden-Finlnd
- NTT Communications from Japan
- Tata Communications from India
- Orange from France
- XO Communications from USA
- Zayo Group from USA
It is not known if AOL Transit Data Network (ATDN) is still a Tier 1 network.
Wait, what… What is Peering?
These networks connect to each other through a process known as ‘peering’. Most traffic needs to go over at least two different top-tier networks in order to reach its destination, and the networks are bridged with peering arrangements. The way this usually works is that each party to the agreement will commit to routing x amount of traffic for the other party on their network, and vice versa. There is usually no money exchanged in these arrangements, unless one side is sending or receiving a lot more data than the other sides.
Large companies can also arrange their own peering relationships. For example Netflix has arranged its own peering and network infrastructure directly with multiple Tier 1 networks so that its traffic is both cheaper and closer to end users on each of the popular US broadband ISPs.
See this Wikipedia page on Peering.
There’s a lot more to read at those pages; this answer gives a general idea, discovering all the details is left as an exercise to the reader. You can ask questions about this subject in the comments below…