Poor NBN FTTN/B design may lead to decades of congestion

July 8, 2015February 15, 2016 Kenneth Tsang Analysis, NBN, Opinion

“With a measly 2Gbps backhaul per node you can forget about 4K Netflix. FTTN is going to be no different to the current Telstra RIMs”

(analysis) Customers on the shiny new NBN FTTN and FTTB networks may find themselves left with slow and congested speeds for decades because of short-sighted network design decisions made by the company.

Analysis of the latest Network Design Rules for the NBN, dated 30th June 2015, reveals that customers may only be able to reach a committed information rate (CIR) of roughly 5Mbps on a fully loaded node – far short of the 25 mbps that popular internet streaming service Netflix says is required for 4K video streaming and also falls short of the Vertigan panel’s recommendation that 50% of Australians will only need 15 Mbps by 2023.

The calculation:

nbn™, the company responsible for building the NBN, currently deploys 4 Point-to-Point fibres from the Fibre Access Node (similar to an “exchange”) to the NBN node where the DSLAM equipment is located. However, it also goes on to say that only 2 of the 4 fibres will be used for connectivity, with the other 2 reserved for “future growth or migration activities”.

NBN's Network Design Document explains 4 fibres will be allocated per NBN node with only 2 in service. — NBN’s Network Design Document explains 4 fibres will be allocated per NBN node with only 2 in service.

Each of the fibres will deliver a 1Gbps ethernet connection back to the NBN Access Aggregation Switch (AAS), totaling to an effective 2Gbps ethernet connection between the node and the Fibre Access Node – or 4Gbps if all 4 allocated fibres are used.

nbn™ introduces an Access Aggregation Switch (AAS) to combine traffic from multiple nodes to the POI

Also according to the document, depending on the DSLAM configuration, each of nbn™’s FTTN and FTTB nodes are capable of connecting up to 384 premises.

Taking all of the above into consideration, in a worse case scenario on a fully-loaded node at peak hour, customers may only reach 5 Mbps if all traffic was distributed evenly:

Number of DSLAM ports	Fibres used for uplink	Entire node’s effective uplink (Mbps)	Committed Information Rate (CIR, Mbps)
48	2	2000	41.7
192	2	2000	10.4
384	2	2000	5.2

Network design wreaks havoc for binge watching season

If a mere 21% of all premises connected to a node starts streaming a 4K stream on Netflix, the node will exceed its capacity.

As many saw with the launch of popular internet TV streaming service Netflix in Australia, telecommunications companies failed to predict the demand of the service leading to heavy network congestion across Australia’s major ISPs.

For some of Netflix’s popular productions like House of Cards and Orange is the New Black, the company releases all the episodes of in their series at once. This results in a brand new network usage “profile” that Australian ISPs and network providers like NBN have seen little of before… where customers continuously watch (binge) and subsequently stream content for hours on end.

If a mere 21% of all premises connected to a node starts streaming a 4K stream on Netflix (21% of 384 at 25 Mbps), the node will exceed its capacity. This will leave zero bandwidth for the remaining 75% of customers potentially connected to the node.

While it can be expected that NBN’s QoS (quality of service) management will balance the load to prevent a small number of customers hogging the entire link, all customers across the board will suffer from congestion issues because of it.

With the increasing prevalence of Internet TV in Australia, the limited design of the NBN FTTN and FTTB networks will have lasting implications on what Australians will be able to do with their Internet connection.

FTTP upgrade path, uncertain

nbn™ has also indicated that they will only deploy 12 fibres up to an NBN node, making it difficult to upgrade an NBN FTTN or FTTB node area to fibre without significant downtime or extensive civil works.

nbn™ is introducing the star topology for the MTM rollout to save money — nbn™ to rollout 12-core fibre to each node, rather than the existing 36 to save money

Assuming a fully loaded 384 port NBN node is to be upgraded from FTTN to FTTP, with 4 fibres already allocated to the FTTN DSLAM for connectivity back to the Fibre Access Node, 8 fibres are remaining to potentially deliver fibre services all the way to the customer’s premises.

However, the 8 fibres will only be capable of delivering GPON services (the FTTP technology that the NBN currently uses) to a maximum of 256 premises (each fibre can be split into 32 premises, 8 × 32 = 256).

Without causing massive disruption to all customers connected to the current node, it may not be possible to transition to FTTP on high-capacity nodes other than by rolling out the network from scratch again.

This means that even if nbn™ decides to upgrade the network, they will likely continue using copper-based technologies for the years ahead to avoid large capital costs again.

(edit) further reading: You can read nbn’s side of the story in their blog post here.

Kenneth Tsang

I'm the author of jxeeno™ blog and co-founder of HSCninja.com. I'm a bit of an #NBN and public transport geek. You can normally find me juggling work and my studies at UNSW where I'm currently completing a degree in Geospatial Engineering.

Matthew Rath

Nice post dude even though it’s just publicizing what most of us knew to begin with, every bit helps i spose.
- tsangk
  
  The difference is that now, we have actual documentation from nbn™… rather than us just talking 🙂
  - Matthew Rath
    
    True, but then i find that rather stupid… independent professionals in Australia and global entities have already vetted that FTTN sucks ballz 99% of the time. Yet we have to wait for the mistake to be made… bizarre doesnt even come close.
    
    Also from my own research the ISAM’s that they’re using are capable of 2 fibres each at 10Gbps a pop, but then i spose i shouldn’t be surprised with the cost saving measures this is the conservative ilk implementing it after all.
Robert

Is there anything to stop NBN from going from 1Gbps to 10Gbps or higher per pair of fibres by changing optics at either end of the fibre run? I’m about to get 10Gbps over 11km out of a pair of fibres, and could relatively easily get to 40Gbps with WDM optics.
- Matthew Rath
  
  Could they do it… yup most definitely, but you’re missing the point, it’s not that they can do it it’s that they DIDN’T do it.
  
  This issue can be examined by looking at the table above containing the number of ports vs node uplink bandwidth (backhaul capacity). In this table you’ll notice that regardless of whatever the maximum number of prems served by a node is the backhaul is ALWAYS 2Gbps.
  
  The mere fact this decision / publication was made in itself shows 1 or more of the following:
  
  a. Lack of insight into consumer demand
  
  b. Lack of understanding as to what contention really is
  
  c. Prioritization of saving capital expenditure over QoS issues which is going to be a big problem considering multiple service providers are all going to be using the same infrastructure (favoritism for ISPs?).
  
  All they had to do was increase the backhaul relative to the amount of users per node and most of the problems could be solved i.e. for:
  
  192 prems – 8Gbps means the CIR would be 41.6Mbps
  384 prems – 16Gbps to maintain this increase from ADSL2 or increase to 10Gbps per line (20Gbps total) for 52Mbps per prem.
  
  The lack of forethought and/or investment here shows that the current gov in charge of NBN is about as interested in improving infrastructure and broadband as an atheist is in religion. With this as established fact i can’t wait to see how my premonitions about the rebranding of HFC come true.
  - Robert
    
    Actually, I’m not missing the point at all.
    
    There’s several assumptions that are required for the existing 2Gbps plan to be bad, primarily amongst them that 20+% of people on a given node will be streaming 4K media concurrently at a time. That’s unlikely at this stage.
    
    My understanding is that NBN will be rolling out around 14,000 FTTN nodes.
    
    1Gbps optics basically come in cereal boxes as promotional items, but you can buy them at list price with no volume for < $50. Let's say $25 each in bulk, and you need four per node, so that's $100 per node, or $1.4m total for the optics.
    
    10Gbps optics aren't as cheap. LR optics are probably OK for most nodes (you'd hope so, ER optics are 5x as expensive), they can be had at around $1500 each without buying in bulk if you get good discount levels, let's say NBN can get them for $1000 each. Even if you lit up each node with only a single 10Gbps fibre, that'd be $2k per node, or $28m for optics. Light up two fibres per node (as per the original plan, because you want some redundancy), and you're up to $56m. Just for optics.
    
    Even if NBN can get their optics at a 50% discount to what I've stated (which is already pretty strong), you're still looking at $14m for a single fibre per node, or $28m for two fibres per node. That's pretty heady stuff compared to $1.4m.
    
    Even at a 90% discount, you're still looking at four times the expense JUST FOR THE OPTICS to light nodes up with 10gbps optics rather than 1Gbps optics, to meet a theoretical worst-case scenario that won't occur for quite some time (and when it does, it won't occur across all nodes, and by then, the prices on optics will have come down substantially, so it will cost a lot less to upgrade than to roll them out today, and the effort to do it will be an OPEX cost, not CAPEX (and yes, that can make a difference).
    
    Whilst I'm absolutely no fan of Malcolm Turnbull's MTM, and certainly not of the way NBNCo and now NBN have gone about things, I don't see this issue being anywhere near as bad as some people are making it out to be.
    
    EDIT: Missed a bracket. 🙂
    - tsangk
      
      Hey Robert!
      
      Thanks for the feedback, you raise some excellent points!
      
      > There’s several assumptions that are required for the existing 2Gbps plan to be bad, primarily amongst them that 20+% of people on a given node will be streaming 4K media concurrently at a time. That’s unlikely at this stage.
      
      Not being critical here, but I’m interested to hear if you’ve been involved in designing and managing NBN ISPs in Australia. I’ve had the (unfortunate) pleasure of writing software to artificially limit and cap speeds (in essence, a crude QoS system) for some smaller ISPs.
      
      From personal experience, network utilisation averages out at 1mbps in non-busy hours… but come after work, a typical sample of users who aren’t speed limited, would exceed 4 mbps at least once (15 min avg samples).
      
      Perhaps using 4K streams as a standard is a bit unrealistic, given 4K tvs are not yet prevalent. What I’m saying above (reaching 4mbps) is driven primarily by P2P downloading and HD videos (Netflix or YouTube). But as soon as you add one or two 4K streams per fully loaded node into the mix, my bet is that you’d start to see problems.
      
      > Even if NBN can get their optics at a 50% discount to what I’ve stated (which is already pretty strong), you’re still looking at $14m for a single fibre per node, or $28m for two fibres per node. That’s pretty heady stuff compared to $1.4m.
      
      Yes, if we upgraded every node to 10GE straight away, it would be costly. But I wouldn’t push for that. From my figures, less than 20% of the FTTN areas NBN is currently rolling out in will have over 250 premises. Only upgrading 10GE on those nodes that have a high-user-count would make most sense to me.
      - Robert
        
        Hi,
        
        Glad to participate. 🙂
        
        If we’re assuming 4Mbps per user, even if 100% of the users were using that concurrently, then on a node with 384 premises attached, you’d be looking at… 1.536Gbps – well inside the 2Gbps limit currently proposed. In fact, with 100% of users streaming at the same time, that 2Gbps limit per node, with a maxed out node supporting 384 households, will deliver a theoretical 5.33Mbps – or if 50% of people are utilising bandwidth, then that’s over 10Mbps each. Go to the 20% of 384 premises in the original article using the node at a given time, and they each get over 25Mbps. As already noted though, most people won’t stream at this rate – 720p HD content uses around 12Mbps – only a little over what’s available to 50% of users at any given time.
        
        2:1 is a pretty low contention ratio on the scheme of things, particularly where ISP network contention ratios are involved. I suspect based on my knowledge of the ISP industry, smaller ISPs especially, that much of that crude QoS software is written to deal with bottlenecks outside of the last mile or node backhaul. Happy to be proven wrong. 🙂
        
        It would be interesting to see the breakdown of the 14,000 nodes as to how many of them exceed the 256 premises limit (given that this gives those same users half as much bandwidth again as the theoretical worst case the arguments thus far have been based on.
        
        I started working in the ISP industry back in 1996, and although I don’t work in it these days, I’ve got a pretty strong connection to many who do. My current job involved planning, designing, implementing and running LANs and WANs for a global multinational, so I do still deal with networks on a day-to-day basis – and the contention ratios on those corporate networks are significantly higher than anything you’ve mentioned (though the use cases are also totally different) – we think nothing of providing 1Mbps per user when it comes to Internet links, and even less when it comes to international MPLS bandwidth, because we know that it’s extremely unlikely that they’ll all be utilising the services at a given time.
        
        Whilst I don’t disagree that there MAY be a requirement for more bandwidth on day one than the currently documented plans allow for, that’s a MAY. What I do also understand is that we’re not talking about 2Gbps (or even 4Gbps if all four fibres are lit up) being the end-game from a technical capability standpoint, and I think there may be some value in being financially prudent at this point, and treating node backhaul contention on a case-by-case basis.
        
        What I would love to see is exactly what metrics NBN will use to measure node backhaul contention, what thresholds will be used to determine that there is an issue to be resolved, and what the resolution plans are (including leadtimes, OPEX and CAPEX costs, etc). I think if NBN added that information to their documentation, then they’d be in a good position.
        
        What I do recognise is that the fact that this information isn’t in this document doesn’t mean that said plans don’t exist. But nor does it mean that they do, either – as I said, I’m certainly no apologist for NBN or the current government’s policies in this area.
      - James Bean
        
        There are a few realities that seem to have been missed here.
        What family house hold has a single device that would be streaming?
        I know in mine with 2 High School kids, if they are not playing games they are streaming youtube HD, doing homework in the cloud, even video howto’s on game design, programming or whatever interests they have (it is endless) or watching their own thing as they aren’t allowed to watch Orangeis the new Black. My 2 primary school kids will be streaming abc for kids or doing homework on mathletics and other apps some of which has streaming content. While I work from home 2 days a week @ 1920×1200 dual screen, and on call after hours if there is an issues, when I am working from home I or my boss don’t want me waiting for things to update or get disconnection issues.
        The assumption that there won’t be that many people streaming 4K is irrevelant, more and more school learning is online, more and more people are working from home, especially people on maternity leave, or even second jobs at night.
        The current FTTN design is based on 1990’s dialup modem values and ways of cookie cutter thinking for what their users want.
        I am on DSL2+ and never have an issue with the above, I had massive issues at a mates FTTN staying reliably online and working at his place during a recent visit, remote link would drop off randomly every 40-60 minutes and that was single screen.
      - James Bean
        
        Issue is cost. The LNP flaunted this $29.6B FTTN as a cost saving step into the future which is now going to cost atleast $56.5B, but this is with 14,000 2Gbps nodes. You say they very well have a plan in place for upgrades for speed into the future, I say at what cost, it isn’t just changing a GBic to make the fibre go faster, that would assume that NBN Co spent extra money now on the nodes to future proof expansion.
        
        Reality is closer to having to replace the gust of each node as well as the equipment upstream at the exchange as its backhaul will be saturated as well.
        
        The costs keep piling up on the FTTN when FTTP would have been $66B-$70B and would never have this issue or future upgrade costs let alone the devastating costs of replacing nodes when massive floods & fires hit our states.
        
        The LNP seems to stick its head in the sand about this, the 2011 floods in QLD would have taken out several hundred nodes and the infrastructure would have been down for MONTHS while they rebuilt. In reality when the water subsided all the exchanges were on high ground, phone and internet for all homes and businesses unaffected by the flood waters were back online FULLY in 2 days.
        
        With FTTN, normal life for an entire community of people & SMB is going to take months not days to start getting back on its feet.
        
        The thing is, Telstra/Optus only provission enough bandwidth to their towers as the load required in the area + 15% (for people travelling through). So nodes go out, everyones landlines go down with floods taking out hundreds of nodes for months, the load on the mobile network is going to be over capacity, calls aren’t going to go through, internet is going to be massively congested and people/SMB are going to get massive bills trying to live & work day to day. Try to make insurance pay for it and your insurance policy cost now goes up.
        
        So the LNP cheaper option puts every single Australian at risk and is a step back to the 1990’s.
      - Chris Pitman
        
        Like cyclone Debbie and the Rockhampton floods.
    - Tipple
      
      You’re only looking at that upgrade cost though, what is the cost to deploy the actual nodes?
      
      If it’s say $100K for everything in a node and getting it installed and connected (That’s $1.4B for 14000 which if the whole MTM is over $40B like they say seems well on the cheap side considering how much of the population it will service) then adding another even $2000 to the cost is nothing. What’s spending another $58M on it to make it that much better if it’s already costing $1.4B or more?
      
      It’s like saying it’ll cost $800 to get a clock in a car which would seem absurd to pay if it was a $15K hyundai but if it’s a $500K supercar then paying $800 to add that supercar clock (while still silly to me) is nothing considering how much it increases the total cost of the car.
    - Matthew Rath
      
      Firstly i wasn’t even referring to a theoretical worst case. To stream 4K video adequately even using the latest codecs you require around 100Mbps per stream that is unless you’re fine with losing frame integrity in high action / motion sequences (sports / blockbuster movies).
      
      Thus the 40 – 50Mbps CIR rates were not indicative of that 4K worst case scenario. They are however consistent with the current multidevice per user paradigm that is already prevailant in society. To stream 1080p you require around 8Mbps bare minimum up to 25Mbps if you want to maintain integrity.
      I don’t know about everyone else but i already have a fair few devices present in my home capable of 1080p:
      
      2 x TV’s
      2 x phones (other 2 are 720p)
      2 x desktops
      2 x laptops
      4 x tablets
      
      If 5+ of those devices are active at any one time (usually are) connection is maxed easily. Just as Bill Morrow said at senate estimates “25Mbps is not enough for my family”.
      
      ++++++++++++++++++++++++
      
      Secondly cost is relative. On average the annual tax collection over all levels of government is around $380 billion (not including the last 2 years of austerity which would hold some discrepency).
      
      Furthermore this is a project being undertaken over the period of at least a decade bringing this up to $3.8 Trillion.
      
      Putting this in perspective it means the $56 million represents 0.00147%
      
      Lets go even further shall we? Even the conservative coalition prices a full GPON NBN at around $70 billion… lets blow it out to $100 billion just for shits and gigs.
      
      This still only represents 2.63%… for a comms infrastructure that would be infinitely more robust and upgradedable… but i digress a bit here.
      
      $56 million is only for populating all 14,000 nodes, as others and myself have stated 10Gbps links would only need to be used where contention is maxed which would also cut the price somewhat.
      
      ++++++++++++++++++++++++
      
      Thirdly I too would share your view that it’s not a huge issue, if they weren’t going to sell chunks of it off to the highest bidder as they have done with so many other public assets.
      
      Is this for certain yet? Nope just my suspicion but given the record of the past 2-3 decades + what i’ve seen so far, it’s seeming more and more likely that this is the end-game for the current office.
      
      If it doesn’t happen we can hope for upgrades, if it does happen we’re screwed again by telstra 2.0
    - C’est la même
      
      $1500 is a trival cost compared to labour maintaining such systems, when spread across multiple users – 1500/192 = ~$8.
      
      To not use higher bandwidth equipment is simply short sightedness – it will end up costing more money as the old/inadequate equipment is inevitably removed and replaced with the necessary equipment.
      
      If the industry is cutting corners like this, they deserve all the criticism they are getting.
- Mark H
  
  Where are you in Australia to be able to get 10Gbps…
  - Robert
    
    I don’t mean on the NBN. I mean over fibre. I’m provisioning an 11km fibre run at present over which I’ll easily get 10Gbps, for a total of $1500 per end for optics.
David Havyatt

Why are you surprised?

Extracts from the (so-called) Strategic Review:

“The largest driver of growth is, and will continue to be, traffic from streaming video online, which is forecast to grow by 29 percent per annum to 2017” P.78

“The multicast forecasts in the Corporate Plan appear to overstate the demand and availability of premium IPTV content in Australia.” P.59

PS Think part of the issue isn’t just the design of the equipment on the fibre but the capacity of the exchange side o the node. More worrying is NO deployment of extra fibre for – FTTdp in future, any consumer buying FTTP. Design compromises to achieve pre-determined cost numbers.

Maybe I need a follow-up to http://www.afr.com/technology/web/nbn/coalition-policy-shift-creates-nbn-co-mess-20150703-ghxzgn
Owen

The decision to reduce contention rate on congested nodes surely won’t be judged with consumer demand in mind. It will be more about ensuring the absolute minimum service to as many people as possible, but only when the population/subscriber density for a given node will “guarantee” that minimum subscriber base.

In short, I will be surprised to see anyone consistently receiving better than ADSL2 speeds from FTTN NBN services during the lifetime of the network – or at least while the status quo is maintained.
Hans van Zitvlak

This is misinformed and assumes NBN couldn’t simply upgrade the upstream interface, which they can with easy replacement of SFP modules. If every network termination card has four upstream interfaces, each capable (with the right SFP module) to run up to 10GE each, that’s a lot of capacity for ~200 node users. Srsly how many 4K streams you gonna run?
- tsangk
  
  > This is misinformed and assumes NBN couldn’t simply upgrade the upstream interface, which they can with easy replacement of SFP modules.
  
  I’m well aware that nbn can upgrade their upstream. I’m also well aware that there are 4 interfaces that support 10GE each.
  
  The crux of my concern, (and perhaps this was not clearly-enough stated), is that with 2x1GE locked in as the default configuration, the company would be unlikely to “upgrade” any nodes where congestion becomes an issue.
  
  They simply do not have any incentive to upgrade in the future because they aren’t servicing end-user customers who are complaining about the slow speeds.
  
  Instead, they are serving wholesale customers who have signed the WBA where any node congestion for TC-4 traffic is not an issue that nbn needs to address.
  
  It is definitely not an issue if they upgrade (even just by adding another 1GE) to a high-usage node. But when the nbn’s MTM model is all about cost minimisation, there is evidently no compelling reason for them to perform any upgrades.
Ian

Um… Am I reading this correctly.. Unless Australia gets immediate and broad spread access to current and future HD entertainment, we’re all doomed? Surely there are other drivers and national needs than entertainment, or is that how we now define and pursue our national vision?
Pretty simple fix then… Healthy premiums for those who ‘need’ access to HD entertainment to boost the capital pool, and cheaper prices for those who ‘don’t need’ 4K Netflix or whatever.
I just don’t see why a pressing critical need for the nation is broadspread access to 4K Netflix. Those who want it can pay for it!
electroteque

Great for productivity, business and the economy. So even if you pay the $5000 to extend the fibre you are still met with congestion.