A major area outage has me thinking about resilancy options in DAA networks

cypherstream

On our local community social media page and neighboring community page there are tons of complaints about Xfinity services having a complete outage all night with an ETA that keeps changing until sometime on 2/4 by the afternoon.

All of these neighborhoods are new DAA RPHY networks. First my neighborhood which had Comcast come in and build a new system in 2022 to now - all down. All 2000+ homes passed. The neighboring area was originally the 750 MHz SA / Aurora nodes but they received the Genesis upgrade and all Harmonic RPHY nodes around the time (and slightly after) my area was newly built. This whole area is out too, as well as the mountain all the way to Albright college. The only other news is in the City of Reading on the Fire alerts page, a fire that took out wires in the city, but I guess that could be the issue if its a major fiber trunk line. But only one path?

I manage quite a few Comcast business accounts and all accounts west of the city of Reading are operating normally. Its basically a large outage east of the city. This leads me to question how do you build resilience and redundancy in DAA networks?

If its indeed one path, why not an alternate to strategic neighborhood / township pockets?
In a DAA node, can you leverage two SFP+ modules and do an MLAG back to TWO 10G switches in the hub? I ran Arista switches for years (still have them at a datacenter). Everything we did to them were MLAG, so if one switch went down, the other leg to the servers and storage were still up. Could 48 nodes be tied back two two 48 port Arista switches, last mile in an MLAG configuration to each node, then the 100G uplinks tied to an upstream fabric? This would all but eliminate switch failure, or switch maintenance. Not that Arista switches go down (ours were up for 5+ years and only rebooted to be relocated to another datacenter).

But here you could have pockets of East / West trunks into lets say a township. Then along that route you could backfill and mlag into nodes. Worth the effort or no? Ok if thats too much work, what about an MLAG from two fibers within the same sheath of cable? That would eliminate the switch / SFP failure aspect. Maybe the Pebble node is just a simplified layer 2 device, and it can't use link aggregation? Maybe it would create a loop and your stuck with the headache of spanning-tree?

Just spinning some wheels here. Curious if any like-minded people made it to this forum that doesn't mind talking about network design a bit.

havaloc

This is slightly tangential, but a few years ago a train in my town caught a drooping wire and zippered around 35 poles, and Comcast was down for quite a while. I think that's why Comcast is emphasizing storm ready and the like, it's probably cheaper redundancy wise to use cellular than to do as you suggest then have multiple trunks feeding a node, etc, especially if most of the pocket is residential. Unfortunately things happen and long gone are the days of the Bell system being bomb proof, etc.

cypherstream

Well those users (2000+) according to the outage map claim now restore by 8PM tonight. If it’s 8PM that will be an outage of 24 1/2 hours.

Cause: equipment damage due to fire.

There is one section of cable on fire from a power line falling on it in a video in our local Fire Alerts of Berks County page. This 100 ft section of burned line seems to really cause quite a large outage, even for those not in that area or indirect path of that area.

Not sure if you'd have access to see this

https://fb.watch/xyfxlWDCta/

cypherstream

cypherstream service restored right about 4 PM.

Outage window 7:30 PM 2/3 - 4 PM 2/4.

EnasYorl

cypherstream The DAA from the Secondary hub to Primary is protected, but looks like a main feeder from the Secondary hub out to all the nodes in the field was damaged in multiple places. The reason your outage is so long is Comcast cannot do any work till all power work is done first then they are allowed to go in and string in a new fiber and splice it up. It if was just a squirrel chew or fiber cut only you would have been up in 3 to 6 hours depending on the size of the fiber count. Most power work would take 8 to 10 hours as they have to run new wires as well as the other wire has spot welding and other damage in it that most companies will have to run a new run then power it up. So based on that video I am not surprised it took so long.

cypherstream

EnasYorl thanks, I knew you would have some insightful information.

I’m surprised my neighborhood was in the outage group.

The fire was about 1518 Cotton St. Reading, PA. Our fiber feed comes down Church lane to oley turnpike rd. Church lane is buried, but it’s fed from Dautrich Rd to Butter Ln. I thought the fiber came from around Antietam Lake but I guess it does either down Exeter Rd to Carsonia or Freedensburg Rd down to the city…

You never know the interesting route it takes. That whole area was out (Mt Penn, Pennside, Exeter, East Reading, Lower Alsace). Big outage.

All of our Comcast sites in West Reading through Robesonia were good. I know you are not from this area but Google maps on these names probably gives you enough OSINT to pinpoint the where.

There’s a lot of talk online how one wire all the way down in the city could affect us all the way out here. Wonder if it was a big count cable… 288+?

cypherstream

Someone on Facebook said 132 nodes were fed from this line.

EnasYorl

cypherstream So 132 nodes muxed would be 7 fibers for DS de-muxes and 7 fibers maybe 8 for US muxes from the area back the secondary hub. I don't know where the secondary hub is but maybe a mile or two from the fire would be my guess. Just bad luck in this case as you move down the line the fiber fan out like branches of the tree. .

So even though it's 192 or 288 count fiber allot of other fibers might be Cell towers and Metro E circuits. But it just take long time to splice all the fibers with light and dark as well.

cypherstream

EnasYorl well there were people having issue with their phones too. Either everyone clogged them up without Comcast being available, or some towers were fed by Comcast.

I know a friend wasn’t able to send text messages until he left the area.

Amazed they can do all that on 7 or 8 strands of glass. Is that with CWDM or DWDM? Keep in mind that whole area was General Instrument (1997-2011) then Aurora nodes (2011-2024), and now most recently Harmonic. So there must have been spares to cut over to RPHY strategically.

The hub is at 3101 Papermill Rd. Wyomissing PA 19610.

The main headend is at 1555 Suzy St. Lebanon PA 17046.

EnasYorl

cypherstream 20 Channels or 40 Channels DWDM depending on optical loss and distance from Secondary Hub where DAA switch located. CWDM is pretty much avoided these days as 100Gbps not supported.

cypherstream

EnasYorl According to Google Maps and "avoid highways" checked off - its 6.4 miles from the secondary hub to where the fire was.

If I add some destinations to the google maps route to mimic the path I think it goes into my neighborhood, its 13 miles to the nearest node where I live.

cypherstream

Years ago, late 90’s, the headend was 400 Riverfront Dr. Reading. They moved the headend to 3101 Papermill Rd when they cut people over to the 750 MHz system. They still used much of the same fiber paths through the city, just built onto it I guess.

Riverfront drive is now a warehouse and Comcast office, but all the techs at Papermill.

All of our zip, 19606 is served by this one fiber trunk that caught fire.

Storm Ready WiFi is surely an option, as long as the local cell tower isn’t fed by Comcast Metro E. A lot of t-mobile towers use Crown Castle. They have fiber along this road too which may also have melted.

You never realize how vulnerable a network can be until you find one of its main arteries.

cypherstream

cypherstream Oh at the hub, do you think they just use channelized optics and terminate right into the switch, or do they come into a mux that breaks everything out, or an ROADM before plugging into the switch?

EnasYorl

cypherstream Yeah you can go 25 miles (40km) thru a 40 channel set of muxes with DWDM with no issues and there are 120km 10G optics out now as well. Dispersion becomes more of an issue. 18.3ps /km.

EnasYorl

cypherstream yes the DAA Swtich has DWDM SFP+ in each port feeding a Mux in a rack that hits a fiber heading out to the field and you can break that into smaller groups of muxes like 4 or 8 channels and use the expansion port to send the light down the road farther.

fmd

Comcast also has a coherent muxponder (CMP):

Last year they presented a small version of it, which sits is a regular node housing (extension site).

cypherstream

So here with 132 nodes down from this fiber burn, were looking at 2 48 port switches and 75% of a third switch with no link on their ports. Each switchport has a unique channelized dwdm sfp+, all linked in engineered groups of X to a mux to get them out a single fiber. Then out in the field that fiber is split to the nodes its designated for and in the node, a matching dwdm channelized optic is installed.... I guess I'm getting this right?

Where do ROADM's come into play? Further up? Enterprise wavelength and ENS services? CRAN?

Heres a quick and dirty I threw together in 5 min in Visio. Not all the right shapes, didn't spend a ton of time on it. Am I in the ballpark? Its prob how I would envision the last mile design.

EnasYorl

cypherstream It's pretty close, but you can have maybe 10 nodes on a single fiber. ROADM is primarily used to transport the 100G rails to the Commercial Edge Routers and the DAA 100 G uplinks (diverse and protected) back too the Primaries that have the PPOD and AR routers.

fmd

Comcast has ROADMs at the larger sites, where many nodes connect back to. Fx at the DAAS hub site. At large CMP sites they are using a variant, the Bidirectional ROADM, BROADM.

cypherstream

EnasYorl nice. Thanks for the good technical discussion. I’m impressed how over the years we’ve been able to leverage more and more on a single wire, be that coax with better order modulations and wider frequency bandwidths, or fiber with more wavelengths and closer spaced lambda’s.

Really incredible. If you remember when they build out my neighborhood, they put Commscope 24ct fiber according to the cable jacket. There’s 3 nodes today, but knowing how much can be muxed onto a single fiber doesn’t make a network engineer nervous about running out.

EnasYorl

cypherstream Well the huge gain i performance was not having to modulate RF/Light in the Headend. Now all the information is Ethernet to the RPD(rPHY Node) and it modulates the whole RF carriers right there. SNR jumps way up as there is no RF to analog LIGHT to RF processing of analog signals anymore. ETHERNET for the win. tne rOLT (ePON) to DAA makes fiber first a choice for new areas.