Top of Rack Design
In the Top of Rack design servers connect to one or two Ethernet switches installed inside the rack. The term “top of rack” has been coined for this design however the actual physical location of the switch does not necessarily need to be at the top of the rack. Other switch locations could be bottom of the rack or middle of rack, however top of the rack is most common due to easier accessibility and cleaner cable management. This design may also sometimes be referred to as “In-Rack”. The Ethernet top of rack switch is typically low profile (1RU-2RU) and fixed configuration. The key characteristic and appeal of the Top of Rack design is that all copper cabling for servers stays within the rack as relatively short RJ45 patch cables from the server to the rack switch. The Ethernet switch links the rack to the data center network with fiber running directly from the rack to a common aggregation area connecting to redundant “Distribution” or “Aggregation” high density modular Ethernet switches.
Each rack is connected to the data center with fiber. Therefore, there is no need for a bulky and expensive infrastructure of copper cabling running between racks and throughout the data center. Large amounts of copper cabling places an additional burden on data center facilities as bulky copper cable can be difficult to route, can obstruct airflow, and generally requires more racks and infrastructure dedicated to just patching and cable management. Long runs of twisted pair copper cabling can also place limitations on server access speeds and network technology. The Top of Rack data center design avoids these issues as there is no need to for a large copper cabling infrastructure. This is often the key factor why a Top of Rack design is selected over End of Row.
Each rack can be treated and managed like an individual and modular unit within the data center. It is very easy change out or upgrade the server access technology rack-by-rack. Any network upgrades or issues with the rack switches will generally only affect the servers within that rack, not an entire row of servers. Given that the server connects with very short copper cables within the rack, there is more flexibility and options in terms of what that cable is and how fast of a connection it can support. For example, a 10GBASE-CX1 copper cable could be used to provide a low cost, low power, 10 gigabit server connection. The 10GBASE-CX1 cable supports distances of up to 7 meters, which works fine for a Top of Rack design.
Fiber to each rack provides much better flexibility and investment protection than copper because of the unique ability of fiber to carry higher bandwidth signals at longer distances. Future transitions to 40 gigabit and 100 gigabit network connectivity will be easily supported on a fiber infrastructure. Given the current power challenges of 10 Gigabit over twisted pair copper (10GBASE-T), any future support of 40 or 100 Gigabit on twisted pair will likely have very short distance limitations (in-rack distances). This too is another key factor why Top of Rack would be selected over End of Row.
Figure 2 - Blade enclosures with integrated Ethernet and FC switching
The adoption of blade servers with integrated switch modules has made fiber connected racks more popular by moving the “Top of Rack” concept inside the blade enclosure itself. A blade server enclosure may contain 2, 4, or more ethernet switching modules, multiple FC switches, resulting in an increasing number of switches to manage.
One significant draw back of the Top of Rack design is the increased management domain with each rack switch being a unique control plane instance that must be managed. In a large data center with many racks, a Top of Rack design can quickly become a management burden by adding many switches to the data center that are each individually managed. For example, in a data center with 40 racks, where each rack contained (2) “Top of Rack” switches, the result would be 80 switches on the floor just providing server access connections (not counting distribution and core switches). That is 80 copies of switch software that need to be updated, 80 configuration files that need to be created and archived, 80 different switches participating in the Layer 2 spanning tree topology, 80 different places a configuration can go wrong. When a Top of Rack switch fails the individual replacing the switch needs to know how to properly access and replace the archived configuration of the failed switch (assuming it was correctly and recently archived). The individual may also be required to perform some verification testing and trouble shooting. This requires a higher skill set individual who may not always be available (or if so comes at a high price), especially in a remotely hosted “lights out” facility.
The top of rack design typically also requires higher port densities in the Aggregation switches. Going back to the 80 switch example, with each switch having a single connection to each redundant Aggregation switch, each Aggregation switch requires 80 ports. The more ports you have in the aggregation switches, the more likely you are to face potential scalability constraints. One of these constraints might be, for example, STP Logical Ports, which is a product of aggregation ports and VLANs. For example, if I needed to support 100 VLANs in single L2 domain with PVST on all 80 ports of the aggregation switches, that would result in 8000 STP Logical Ports per aggregation switch. Most robust modular switches can handle this number. For example, the Catalyst 6500 supports 10,000 PVST instances in total, and 1800 per line card. And the Nexus 7000 supports 16,000 PVST instances globally with no per line card restrictions. None the less, this is something that will need to be payed attention to as the data center grows in numbers of ports and VLANs. Another possible scalability constraint is raw physical ports – does the aggregation switch have enough capacity to support all of the top of rack switches? What about support for 10 Gigabit connections to each top of rack switch, how well does the aggregation switch scale in 10 gigabit ports?
Summary of Top of Rack advantages (Pro’s):
-Copper stays “In Rack”. No large copper cabling infrastructure required.
-Lower cabling costs. Less infrastructure dedicated to cabling and patching. Cleaner cable management.
-Modular and flexible “per rack” architecture. Easy “per rack” upgrades/changes.
-Future proofed fiber infrastructure, sustaining transitions to 40G and 100G.
-Short copper cabling to servers allows for low power, low cost 1oGE (10GBASE-CX1), 40G in the future.
-Ready for Unified Fabric today.
Summary of Top of Rack disadvantages (Con’s):
-More switches to manage. More ports required in the aggregation.
-Potential scalability concerns (STP Logical ports, aggregation switch density).
-More Layer 2 server-to-server traffic in the aggregation.
-Racks connected at Layer 2. More STP instances to manage.
-Unique control plane per 48-ports (per switch), higher skill set needed for switch replacement.
credit to source
