Judge Koen said he had decided "to recuse myself from the trial" to avoid any risk of compromising the perception of justice.
His rejection of Zuma's bid to oust Downer could have been held against him in the future, "when the issue of whether Mr Zuma has received a constitutionally fair trial will arise for determination," Koen said.
Stepping aside "is what the sound administration of justice, the requirements of the constitution and my conscience dictates," Koen told the court in Pietermaritzburg.
"The integrity of the judicial process must be protected against any reasonable taint of suspicion so that the public and litigants may have the highest confidence in the integrity and fairness of our courts," he said.
Judge Koen last year dismissed Zuma's bid to force out the prosecutor, Billy Downer, whom he accused of leaking confidential medical documents to the media.
In December the Constitutional Court backed Koen's decision, concluding that Zuma's application to remove Downer had no reasonable prospects of success.
The decision by the top court cleared one of the last legal hurdles for the long-running trial to get underway.
The situation has been further complicated, however, by a bid by Zuma to force Downer out through a private prosecution.
Zuma faces 16 counts of fraud, graft and racketeering over the purchase of fighter jets, patrol boats and equipment while he was vice president in the late 1990s.
The embattled former head of state's long-awaited trial ran into a string of legal delays after starting in May 2021.
In between his defense team secured a postponement on health grounds.
Zuma, 80, was president from 2009 until 2018, when the ruling African National Congress (ANC) forced him out as graft scandals besetting his government brewed into a political storm.
In July 2021, he was given a 15-month jail term for contempt of court after refusing to testify before a panel probing financial sleaze and cronyism under his presidency.
That case is separate from the arms scandal, in which he allegedly took bribes from French defense giant Thales, which has also been charged with corruption and money laundering.
Both Thales and Zuma deny any wrongdoing.