Joshua has won all of those previous bouts, 21 by knockout.
Eddie Hearn, who promotes Joshua, is concerned with the level of risk involved in the contest.
The 29-year-old Joshua is 6-6 and generally fights in the mid-240s, so Miller will likely have more than a 60-pound weight advantage. "Now he goes to the Mecca of boxing, the iconic Madison Square Garden, where the great heavyweights have performed - Ali, Frazier, Louis, Marciano, Tyson", Hearn told Sky Sports.
Instead, Joshua will make his debut in the United States, having been keen to grow his profile across the pond, though he could not hide his disappointment at missing out on fighting Wilder.
"June 1st I am heading to The Big Apple and I plan to embrace the culture and leave with an appetite for more", said Joshua.
'I would have loved to fight at Wembley and brought you guys the undisputed championship of the world but I'm looking forward to the new challenge. "I will leave nothing to chance and plan on dismantling Miller in style to make my mark".
Jarrell "Big Baby" Miller certainly got under the skin of Anthony Joshua when they met last summer. will he try it again before June 1?
"Miller is a huge personality and huge is stature - he is about 22 stone".
New York's Barclays Centre or Madison Square Garden is expected to host the clash, which will take place in either late April or mid-May, although the date and venue have yet to be officially confirmed. New Zealand's Joseph Parker is the only fighter to take Joshua the distance. "That punk AJ is standing in the way of my dreams and on June 1 he's getting run the hell over".
Joshua is looking to unify the heavyweight belts but negotiations broke down past year with WBC titleholder Deontay Wilder, who ended up fighting Tyson Fury in December. "The good news for Miller is that he gets this fight in his backyard".
Joshua is one of the most popular fighters in the world and has drawn crowds in excess of 80,000 in the U.K. Bringing him to fight for the first time in the USA will increase his name recognition here and make him into more of a global superstar.
Fury insists he stayed true to his promise to donate his entire purse from the first Wilder bout to charity but hopes that it's known he did not do so just for the positive publicity.