In a match of three distinct phases, South Africa dominated the first quarter to build a 17-0 lead before a 52,790 crowd on a cold night in the South African financial capital.
England recovered to score twice and close the gap to just three points – 17-14 – by half-time as they sought to end a four-match losing streak started in the Six Nations Championship.
The first England try came when South Africa were down to 14 men after Arendse was sin binned for a deliberate knock-on.
After two more tries from the Springboks in the second half, Coles barged over and the Smith conversion left England trailing 31-21 entering the closing stages.
Hopes of a dramatic comeback by the visitors were dashed after Freeman banged his shoulder against the head of South Africa full-back Damian Willemse.
Three minutes later, Pepper joined Freeman in the sin bin for an early tackle.
South Africa took advantage of their numerical superiority with Marx and Dixon scoring and Kolbe, given the goal kicking duties ahead of fly-half Manie Libbok, converting both.