Segner has undergone minor surgery to wash out an infection in his ankle joint and will return home to continue his recovery.
All Blacks Head Coach Dave Rennie confirmed an injury replacement would not be called into the squad at this stage.
Segner made his Test debut against Italy in Wellington in July and has featured prominently during the tour of South Africa, including a standout performance in the 50-19 win over the Bulls in Pretoria.
The All Blacks now turn their attention to the second Test against South Africa in Cape Town, as Rugby’s Greatest Rivalry continues this weekend.