A major area outage has me thinking about resilancy options in DAA networks

cypherstream · 2025-02-04T14:45:14+00:00

On our local community social media page and neighboring community page there are tons of complaints about Xfinity services having a complete outage all night with an ETA that keeps changing until sometime on 2/4 by the afternoon. All of these neighborhoods are new DAA RPHY networks. First my neighborhood which had Comcast come in and build a new system in 2022 to now - all down. All 2000+ homes passed. The neighboring area was originally the 750 MHz SA / Aurora nodes but they received the Genesis upgrade and all Harmonic RPHY nodes around the time (and slightly after) my area was newly built. This whole area is out too, as well as the mountain all the way to Albright college. The only other news is in the City of Reading on the Fire alerts page, a fire that took out wires in the city, but I guess that could be the issue if its a major fiber trunk line. But only one path? I manage quite a few Comcast business accounts and all accounts west of the city of Reading are operating normally. Its basically a large outage east of the city. This leads me to question how do you build resilience and redundancy in DAA networks? If its indeed one path, why not an alternate to strategic neighborhood / township pockets? In a DAA node, can you leverage two SFP+ modules and do an MLAG back to TWO 10G switches in the hub? I ran Arista switches for years (still have them at a datacenter). Everything we did to them were MLAG, so if one switch went down, the other leg to the servers and storage were still up. Could 48 nodes be tied back two two 48 port Arista switches, last mile in an MLAG configuration to each node, then the 100G uplinks tied to an upstream fabric? This would all but eliminate switch failure, or switch maintenance. Not that Arista switches go down (ours were up for 5+ years and only rebooted to be relocated to another datacenter). But here you could have pockets of East / West trunks into lets say a township. Then along that route you could backfill and mlag into nodes. Worth the effort or no? Ok if thats too much work, what about an MLAG from two fibers within the same sheath of cable? That would eliminate the switch / SFP failure aspect. Maybe the Pebble node is just a simplified layer 2 device, and it can't use link aggregation? Maybe it would create a loop and your stuck with the headache of spanning-tree? Just spinning some wheels here. Curious if any like-minded people made it to this forum that doesn't mind talking about network design a bit.

cypherstream

I just thought of this driving in today... Imagine if our new build was Node+0. I can see how many more DWDM channels that would have taken up on the trunk, thus spilling into more fibers. Imagine if our new build was EPON FTTH? Maybe there isn't enough fiber in that trunk to deliver that. Maybe the holy grail of EPON out of DAA nodes where required makes more sense.

The original area neighboring mine was probably around a Node+6 ish on the Aurora nodes. I've seen 1 for 1 replacement to Harmonic, but not sure if the nodes are split further.

If one Node in my area serves 150 homes passed, what would be a Node+0... 50 homes off a node? We'd have 6 nodes instead of 3 in my development alone. If they built EPON, I'm not sure what the split is, 1:32? Thats even more wavelengths (then fibers).

Then talking about future network optimization, creating a bridge between Freedom headend in New Hanover, PA and Keystone in Reading PA, for resiliency and better peering - I guess that would have to ride this fiber trunk as well, because who wants to run 20 miles of additional fiber optic cable through the city. Unless they connect through a back way via Oley (existing) / Blandon (new) area.

fmd

Comcast goes after the highest possible splits. Up to 1:128 (the current OLTs support a max of 128). A slide from one of their papers:

EnasYorl

cypherstream Well the huge gain i performance was not having to modulate RF/Light in the Headend. Now all the information is Ethernet to the RPD(rPHY Node) and it modulates the whole RF carriers right there. SNR jumps way up as there is no RF to analog LIGHT to RF processing of analog signals anymore. ETHERNET for the win. tne rOLT (ePON) to DAA makes fiber first a choice for new areas.

cypherstream

EnasYorl Yes at least 3db if not a little more in SNR improvement and a very nice flat trace according to the spectrum analyzer app I used plugged into the Comcast tap.

But the thought is NODE+0, NODE+1, NODE+2, NODE+n and homes passed. Obviously, we all thought the smaller node groups the less potential "contention" in the last mile, and less amps to take care of (and eventually upgrade to FDX). But seems a moot point these days.

A 1 to 128 split for EPON doesn't appear to me that much better than a NODE+0, heck node+0 is better if its 50 homes passed. I'm talking once we get into FDX territory.

All bets off if we go further into the future to 25GPON and beyond.

EnasYorl

cypherstream you need to look at the RPD and the rOLT you can add another 10G optic and split the FLM feeding the homes.

fmd

cypherstream I can add that one of Comcast's N+0 design criteria is that HHP must be no larger than 128, so it can fit into a single PON.

cypherstream

fmd if N+0 has a 128 home design constraint, and N+1 or 2 has a 150 or so homes passed node pocket, then I can understand not doing N+0 as the additional fiber budget and more expensive node hardware doesn’t really gain that much of an advantage. With higher output, low distortion modern amplifiers, you are still receiving 44-45 SNR. At least I am. I’m EOL 8 value tap, and on an older SB8200 it was reading 43-45 SNR across 32 QAM, 2 OFDM channels.

On SECV cable, it’s traditional digital return in OM6000 nodes, so I see 40-42 SNR off a 11v 8 port tap, along with a roll off in the high end- acceptable levels still but clearly visible, vs Comcast’s completely flat trace out to 1002 MHz using the same test equipment.

MacLeech

cypherstream The rPHY nodes can be outfitted with more than 1 EPON OLT module, with each supporting 128 homes. Also, the uplink to the nodes are still the same since it's ethernet, there isn't really any change to the fiber budget on that side.

The bigger advantage comes in the not too distant future with coherent PON (CPON) modules being used.
https://www.cablelabs.com/news/cablelabs-issues-architecture-specification-for-cpon-100gbps-single-wavelength-pon
https://account.cablelabs.com/server/alfresco/25300435-87c8-407d-9ab0-722287c7c7a6

fmd

cypherstream Comcast reported that 77% of their mid-split customers are using 4096-QAM upstream (92% use 2048-QAM or above). They seem to be pleased with that.

cypherstream

MacLeech The CPON is exciting. I’m looking forward to seeing that deployed in the real world. Thanks for linking that paper. I’m going to read it later when I get a chance.

With 100gbps on a wavelength, that’s a lot riding on it. Still makes me wonder if it’s worth an East/West feed to a neighborhood.

The fire that took out this stretch of cable took out service to an entire zip code… a large one at that. Both existing areas, and new edge our builds that have occurred between 2022-2024. I don’t know about feeding every node with an east/west feed. What about small fiber hut with diversity running G.8032 ring protection, and put the switching there. Kind of push out the ethernet generation further in the field. I guess real estate, security, power and cooling make that a challenge to break even.

If I take a step back, how often are major fiber trunk lines that take out hundreds of nodes cut up? I think in our area this is a first for Comcast. SECV had a similar one in the 90’s where a tractor trailer crashed with someone into a pole, exploded and the whole pole and everything on it burned up. It was close to their analog headend back when it used to be on Ridgewood Mountain, Rt 724 and 10. I wanna say this was 1997 or so and took out the entire system for a day. That mountain facility is still there but they moved the headend to 6400 Perkiomen Ave. Birdsboro PA. They recently added another few poles and fiber exit points on the property, likely part of their PON deployment.

Anyway I guess in normal circumstances this kind of event shouldn’t happen and it’s business as usual.

fmd

The fiber repair work is also one of the things that Comcast highlights when they talk about coherent fiber in the access network (the CMP). 100 Gbps or more per wavelength using just a single fiber. Less fiber, less repair work.

cypherstream

fmd On page 30 of the CableLabs CPON document, there is a section on Protection and Resiliency. This document helps clear a lot up.

« Previous Page