South Africa's 36-34 win was a timely reminder, then - nearly exactly a year out from the next World Cup - that everyone is beatable.
After the fourth round of the Rugby Championship, the All Blacks have 16 points, followed by South Africa on 10, with Argentina third with eight points after beating Australia who are last with five points.
His All Blacks counterpart Steve Hansen has congratulated the Boks.
"The boys showed guts, and I am so glad for them", said Erasmus.
Captain Siya Kolisi said the win was special, given the many challenges faced by the team in the build-up to the tie and during the 80 minutes.
"When we started planning we thought if there's one way to come back it's to beat New Zealand in New Zealand. We've got to catch up with Australia and England, there are so many teams ahead of us there's no way we can get ahead of ourselves", he added.
"Throughout the week‚ we focused on not switching off at any moment in the game because if you do that they will pounce on you‚" he said.
'When we lost those two games in a row, you do lose some belief but we always targeted this game to turn things around and be real contenders at the World Cup.
"It's a special win, a special day, and I'm so proud of the guys".
"The result is a credit to South Africa".
Willie le Roux leaps into the air after kicking the ball out to clinch victory for the Springboks against the All Blacks.
At half time the Springboks were leading 24-17 with tries from Aphiwe Dyantyi, Willie Le Roux and Malcolm Marx, while Handre Pollard landed three conversions and a penalty.
The Boks are now second on the Rugby Championship table with 11 points after two wins - against New Zealand and Argentina.