The potential drug, which has not been tested in humans and remains years away from possibly becoming available, joins a growing number of male contraceptives in development.
Previous drugs have struggled partly because the bar for side effects is believed to be far higher for men - because they do not risk getting pregnant - as well as a lack of interest from the pharmaceutical industry.
There are currently only two options available for men: condoms and vasectomies.
"For women, right now all the burden of contraception is on us," Melanie Balbach, a pharmacology researcher at Weill Cornell Medicine in the US, told AFP.
"We want new options," said Balbach, the lead author of the study published in Nature Communications.
The team of researchers targeted an enzyme called soluble adenylyl cyclase, which acts as the "on switch" for sperm, said study co-author Jochen Buck, also of Weill Cornell Medicine.
If the enzyme is switched off, the sperm can no longer move, he said.
Across several different tests, the researchers showed that a compound which blocks the enzyme renders mice sperm immobile in 30 minutes to an hour.
The compound was 100 percent effective in preventing pregnancy within the first two hours, dropping to 91 percent in the first three hours, the study said.
After 24 hours, the mice sperm moved like normal again.
The researchers hope are aiming for a single non-hormonal pill that works in under an hour and lasts six to 12 hours, Buck said.
This would be much different to other options under development, such as a hormonal gel currently going through human trials, which all take weeks or months to start and stop working.
No side effects were noticed in the mice. Previous research has suggested that infertile men who had their soluble adenylyl cyclase enzyme permanently switched off had an increased rate of kidney stones.
Buck said this was the result of their enzyme always being off -- which would not be the case for men taking an on demand pill.
The researchers hope to hold the first trials on humans within three years, with a final product possibly up to eight years away, Buck said.
Susan Walker, an expert in contraception at the UK's Anglia Ruskin University not involved in the research, said she was a "little skeptical" a pill would actually make it to market as so many other efforts have fallen short.
But the "eye-catching advantage" of almost immediate effectiveness offered "the possibility of seeing a sexual partner take a pill," she said.
The consultancy Desire Line is working on forecasting the potential uptake of a range of male contraceptive products, according to its founder Steve Kretschmer.
"Initial estimates indicate in the United States that uptake for an on-demand pill which has quick onset of action and 1-2 days duration of action could be about triple that of Viagra when it was initially launched," he told AFP.