Data Center Comms Cabling

In the two data centres that I have visited the cables are all single- mode optical fibres. There is often a coil of excess fibre in the bottom of the rack just above the hole in the floor where they enter.

John

Yes, a floor panel is not installed along the line of racks, this permits cable management and airflow - traditional method of raised floors.

Non raised floors have also need used for decades, but more prevailent in recent years and use cable trays from above into racks, easy as racks also come with overhead cable management access.

not usually any patch panels - unless the customer supplies one for their local network there, usually the fibre (or 2+ if multi homed - footnote: if leasing rack in an internet exchange, they usually REQUIRE you to run BGP and take a peering feed with them) is dropped inside the rack with plenty length, and you connect it direct into your border router, since space and power is limited inside leased racks, its not worth using patch panels, go from server to switch direct usually with cat6 (6a is pointless over the barely 2 metres from any server position to your switch) and usually fibre from switch to router, but often that bit can be ethernet, its up to you as its inside your part of network.

Cheers

Are people still using SFP modules for the fiber links? Or something denser?

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

OK, the cable acting to carry the traffic for EVERY host in the rack to/from a "main switch", elsewhere.

A switch inside the rack to handle the fan-out to the hosts INSIDE the rack? (that switch effectively acting like a patch panel for the rack)

I suspect the "hole" is just a lifted floor tile, if the optical cable is "stored" in the rack? This (hole and service loop) not being an eyesore because:

- it's hidden inside the confines of the rack

- it's a data center and not expected to be aesthetically pleasing (err on the side of serviceable)

The cosmetics aren't an issue, then (see my reply to John).

Similarly, "not an issue".

The router/switch serves the function of a patch panel. My point was, someplace for individual network cables from each host in the rack to interconnect (you're not running 20 cables into/outof the rack for the 20 hosts within)

I was thinking in terms of the datacenter belonging to and serving ONE entity. Like a large business..

I need to reframe my question, then...

A lifted floor tile would be much too big a hole. Its a small hole cut into a floor tile. Maybe 10 or 15cm diameter. Optical fibres are thin and a couple of power cables don't take much space.

John

I thought lifting the tile and leaving it partially in place (to expose a "slot").

So, if the excess cable is piled in the rack, do they "squeeze" the coil to feed it through said hole? (Or, pull it through one foot at a time?)

In any case, everything is hidden soas not to be an eyesore. And, everything is fixed in place so no fear of someone "moving" a rack, casually.

My problem is more on the order of having the switch just outside the rack and cabling it's ports to the servers within. In a "clean" and robust manner.

Anything denser than SFP would be very difficult to keep cool enough. As it is, some SFPs get very hot.

In terms of bandwidth per unit volume (and cost) SFP28 modules are very impressive. I recently bought a pair of 25Gbit/s single mode SFP28s with 10km range for about USD25 each from Amazon UK. They were made by

They are electrically and mechanically compatible with SFP and SFP+. I have tested the 25Gbit/s modules in a 1Gbit/s SFP link without any problems.

John

I'd sell something like that for $5000.

Yes!

We worked on one project that mounts a missile on a 6-axis gimble and twirls it around for some reason. Data goes up and down through fibers and optical slip rings. We used WDM SFPs to move data up and down over each fiber. Fast bidirectional WDM SFPs cost something absurd like $20.

Here's my SFP test board.

Unfortunately, SFPs are AC coupled in and out, but the AGC sorta requires that. The LF response varies a lot. Most will work down to

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

I notice that the traces appear to widen as they separate near the SMA connectors. Do you happen to have any spare boards please? They look really useful for testing.

John

Trying to keep the impedances constant.

I think I have a couple more. Email me.

jjlarkin mainly highlandtechnology etc

I can let you have the files too, and you could order boards from JLC.

John Larkin Highland Tech Glen Canyon Design Center Lunatic Fringe Electronics

Yes, in an unmanaged colo solution, usually SC, but some datacentres use LC, and most these days in these parts at least use singlemode

MPO'smore common in carrier internal networks

Correct, racks have doors, some wire mesh, some smoked glass if you had no access to any racks and just walked around for 2 minutes you'd never know.

switch would, yes. usually you wont have 20 or even 10 hosts a rack, you don't lease by RU - security reasons, you lease full, half or in some cases quarter racks, and each customer gets a fibre from the IX switch to their section.

leasing by 1 or 2 RU is possible but in managed colo, where you dont get to physically access it on your own anytime you want, again, security reasons since you'd have access to every other RU, and in such cases you would only get ethernet, becasue thats a to-server direct-path, no router

- you'll be subject to routing from whoever you rent the server/server space from, ie: a local ISP, and you'd like only get one IPv4 address and possibly an IPv6 /64, no IPv4 subnets, no AS required, so BGP, no nothing.

caveat: you could have a few servers and lease a few RU but by this time the cost will be cheaper to lease a quarter rack.

caveat2: larger IX's are like laws, written requirements that are not movable or even able to be "streched". Smaller IX's or ISP data centres are similar, but the goal posts can often be moved under negotiation.

Ahhh in that case, all te rules go out the window, it would depend, for example, if you have 20 servers in a rack, I've seen plenty of "enterpise data centres" which are essentially "server rooms" that have

I think most of us realised about 20 years ago patch panels only suite fixed-cabling workstations/WiFi/phones/CCTV/Security to the server room.

Patch panels for in-rack equipment, only benefits their manufactuers bank balance :)

I'm trying to address the issue where a rack ("equipment closet") has to terminate scores of cables that run off to various parts of the building/enterprise.

Putting a switch in the rack acts like a patch panel allowing N wires inside the rack to connect to M wires outside (using the fibre as a bridging link)

Imagine running a server(s) for your business. There are drops in each cubicle for the various clients that each employee uses.

The near ends of these cables have to exit the "building structure" (dropped ceilings, inside walls, etc.) and emerge somewhere in the "equipment room". There, to join with cables that connect to your servers (data and voice and ???) likely via switches.

So, you have the issue of getting the cables out of the walls. Then, connecting to <whatever>.

If <whatever> is housed in a rack, do you run each of those individual cables into that rack?

Do you site a switch outside the rack to terminate the majority of the cables and add the few singletons from rack-housed servers?

Do you terminate the building cables on a "patch panel" and the connections into the rack on a similar panel and then connect the two with patch cords (which allows the rack to be moved from a fixed location)?

Telco wiring used to just terminate on a fixed panel mounted on a wall. Any kit located in the equipment room (e.g., PBX) was similarly terminated. If you wanted to remove/service that kit (PBX), you disconnected it from the punchdown panel (usually with 50 conductor cables terminating in D-shells).

The data center descriptions seem to suggest moving a rack is just something that never happens. You *gut* it, in place, if need be. So, the *few* cables that enter into it (via floor) can just be disconnected during that event.

These drops terminate in a switch, somewhere.

It's what we do, and by we, I mean the industry, in internet exchanges and carriers, because often that hole is only one of any entry points for power and interconnects, its more about air flow throughout the rack.

Since you have in one of these thrreads said its a single floor cenario, and as alluded to in previous post, yes, each wall outlet is hard terminated from its mech insert (6p6c / called RJ45 or keystone jack in some countries), cabled up inside the wall into and through the ceiling to above the patch panels rack, where a small cutout is made in the ceiling next to the wall, down into the rack and into a patch panel, made neat by using 2 peice pvc duct, you then run patch cables from each patch panel port to switch.

Any servers locaed in serer room get patches directly from server to switch - no patch panels involved.

covered above

All terminations should be done in the same rack, switches and patch panels are only 1RU in height, you'll need a few small OM4 cables to link the switches together, 150 odd endpoints wont be a problem with arp storms so you can keep it simple.

racks should never be moved, if you have over 100 connections thats especially so, so plan your design right the first time - leave space around it in case you need to add racks, or move it over one - always, always leave about 2 metres extra cable coiled in the ceiling, this permits those "gotta move it a few feet this way or that way)

not sure what you call D-shells, must be region specific term, I assume you mean telco50's (also called amphenols), so you have analog phones and not using SIP? so what do we have here, 144 pc end points plus 144 phones or 72 pcs and 72 phones?

correct

hang on, your a single floor, with only 1 or maybe 2 racks, you wont be using a raised floor, your drops are from the ceiling into hte rack, therefor your server room only need to be small so your racks wont need doors orif so use mesh doors and a local aircon unit not a specialised HVAC unit intergrated with fire smoke detection and suppression, that'll cost more than a yewars rent and a raised floor which is not cheap either, along with certifications especially if you live in earthquake prone regions.

the drops as above termiate into a patch panel with is jumpered (patched) into switch

The installer has to sort out the tradeoffs of installing multiple distributed "concentration points". The downside is you need multiple locations in the living space that you can set aside for these uses (consider power distribution as well). In existing buildings, this is usually hard.

Here (alpha site), everything is on a single floor and the "drops" (nodes) are so close together that it is impractical to have multiple sites (switches).

So, the patch panel is *in* the rack; there are MANY CABLES traveling into the rack and the rack is effectively "tied" to that location.

Even if in another rack?

I don't run normal IP so most of those issues are... "different".

Here, there is no "ceiling space". The ceiling is effectively the underside of the roof (roof substrate sits on ceiling joists to which the interior ceiling is fastened). Our "attics" are 12" tall and 16" wide. :>

roof (outside of building)

---------------------- | | | joists

---------------------- ceiling (interior of building)

Think, instead, of an automobile (late model). You likely have 50 or more ECUs distributed around the vehicle:

- there's one in each headlight

- one in each door (switches, door solenoid, sensors, widow motor, etc.)

- several in/around the engine

- one in the transmission

- infotainment unit etc. These are each very different devices with specific functions. They are interconnected via a bus (CAN bus) instead of a switch (star).

Conceivably, you could connect the transmission to the connector for the passenger side door. It would still work. Much like you could swap two workstations and expect the comms to sort themselves out (adaptive switch).

In my case, the devices are cameras, speakers, displays, HVAC system, alarm system, telephone system (FXS/FXO), weather station, access controls (door locks), irrigation, genset, solar collector, water heater, stove, refrigerator, etc.

In theory, one could physically move any of these devices to take advantage of some other cable drop. FOR A CONVENTIONAL SWITCH.

But, this poses serious design flaws. Much the same way that having every data structure in a program accessible to EVERY line of code allows faulty code to do things it shouldn't. And, malevolent code to do even worse!

This ^^^ is possible because the access is unauthenticated and can be injected ANYWHERE in the vehicle. There's no concept of a "secure node" (think ttys(4)) that merits special trust. And, no way to tell the "door lock" that the message is originating from the "key detector".

The same is true of the ports on a switch. A program running on the device connected to port 4 could issue a command that SHOULD have come from the REAL program on port 8.

I restrict the transactions that can originate at any given device. E.g., its inconceivable for the HVAC system to issue a command to unlock the front door. BECAUSE THE SOFTWARE THAT CAN ISSUE THAT COMMAND ISN'T RUNNING ON THE HVAC CONTROLLER! So, if such a command came from the HVAC controller, I would know the controller to be defective or compromised. In either case, take it offline.

So... if the user (homeowner) has to rewire the switch (possibly at the behest of the switch!), the switch needs to know which physical wire is being connected to each port. Swap the cables on port 1 (securely routed) and port 2 (insecurely routed) and the system will now treat the device on port 2 as untrustworthy.

Additionally, the system needs to power down devices before they are un/re-cabled. And, arrange for any "work" that a node is doing to be transfered to another node (or, paused/aborted if that isn't possible).

So, it's not just a casual activity of plugging and unplugging patch cords. Instead, those actions have to be *directed* (by the system).

Equipment sits in the garage in a secured cabinet (physical protection against the above video sort of attacks). It lets me move some of the heat out of the house and lets the kit be located closer to the backup batteries, as well as not as much of an eyesore. (we don't have basements and a PoE switch that big isn't easily hidden in a closet)

To make that easier for the homeowner (working inside a rack isn't easy -- especially if you are blind or physically handicapped), I plan on putting two panels -- "switch" and "field" -- with ample space to run the interconnect cables. If the rack isn't intended to be movable, then there's no reason NOT to make it a permanent installation!

So, the terminations of the switch's ports and the field drops can similarly be "fixed" in their locations. And, the panels don't have to be cramped (you can get 144 ports in ~3U but good luck managing the cabling -- especially if you are not an IT professional!)

WTF, you've gone from a business - medium size with 144 drops all workstations so close together, along with 144 FXS so your building out a network that doesnot use SIP, but is preparted to pay a lot more for gateway, to a home with a blind person doing IT, I think I'm talking to an AI - which is far more A , than I, or you're trying to set up a call centre for scam artists, or since yu want to give a drop to stove and move it around, possibly bad drugs.