 Fumio Kyuma said he was sorry for the trouble he had caused |
His apology, and a public rebuke from PM Shinzo Abe, failed to quell anger over his remarks that the bombs on Nagasaki and Hiroshima were inevitable.
The resignation comes at a sensitive time for Mr Abe, who faces national elections at the end of July.
Mr Kyuma is being replaced by national security adviser Yuriko Koike.
Mr Kyuma is the second minister to resign under Mr Abe. A farms minister, also hit by scandal, committed suicide in May.
Mr Abe's 10-month premiership has been hit by a series of scandals, including one involving missing pension records which has angered voters.
Polls show his support has dropped to its lowest level since he took office.
Election battle
"I regret that my comments have caused trouble. I am very sorry," Mr Kyuma told reporters on Tuesday, adding that Mr Abe had accepted his resignation.
His resignation came as the outcry over his remarks showed no sign of diminishing following his earlier apology and public dressing down by Mr Abe.
He was criticised by some of his fellow ministers, and faced a formal request for his resignation by the opposition later on Tuesday.
 The bombs on Hiroshima and Nagasaki killed 360,000 people |
Mr Kyuma, who represents Nagasaki in parliament, said in a speech on Saturday that the bombs on both his own city and that of Hiroshima in 1945 were "something that couldn't be helped".
He said that, while the bombs caused great suffering in both cities, the US must have thought they "could prompt Japan's surrender, thus preventing the Soviet Union from declaring war against Japan".
Japanese leaders rarely comment on the use of atom bombs against Japan, for fear of damaging ties with the US.
He apologised a day later for giving the impression that he lacked respect for the victims of the bombings.
And he was told by Mr Abe on Monday that he must "strictly refrain from making remarks that cause misunderstanding".
National security adviser Yuriko Koike would succeed Mr Kyuma as defence minister, the government confirmed.
Poll countdown
Mr Abe now faces an uphill battle ahead of upper house elections on 29 July, after weekend opinion polls showed support for him had dropped to under 30%.
A heavy defeat for the ruling coalition in the election could lead to calls from within his party for Mr Abe to resign.
More than 360,000 people died as the result of the two atomic bombs dropped on Hiroshima and Nagasaki in 1945, in the last stages of World War II.
