Holding R does NOT restart the ante.

It restarts the whole run, from ante 1, with the same deck and same stake but with a different seed.

There's a "zen mode" mod that allows you to redo an ante when you lose, it may have reworked the hold R behavior and you stumbled upon someone discussing that. But yeah normally it just starts a new run, like Malogoss said.
I assume savescumming with backups of the save file would work, though I haven't tried it and it seems like a lot of effort to do it routinely enough to really be helpful.

That zen mode mod, in case you're interested: https://github.com/Amvoled/Taikomochi
Uses the Steamodded modloader ( https://github.com/Steamopollys/Steamodded ). No idea if it works in the current version of the game though, has it been 5 months since the big update came out? Balance patches are less likely to break stuff.