Know the chemistry first
CR / BR lithium coins are usually about 3V.
LR alkaline and SR silver-oxide buttons are about 1.5–1.55V.
A correct voltage on the wrong chemistry is still the wrong battery.
Multimeter method
Set the meter to DC volts. Touch probes to the cell (flat marked face is typically +, but confirm).
Compare to the expected voltage for that code. A CR2032 well below ~2.5–2.8V is usually tired; treat readings as guidance, not a guarantee.
Remove the cell from the device when you can so you are not measuring through the circuit.
When replacement is simpler
Coin cells are cheap. If the device page names CR2032 and behaviour matches a dying cell, fitting a fresh correct cell is often the practical test.
Watch for model-specific issues (bitterant coatings, O-rings, re-pairing) on the device page after you change the cell.
Do not
Do not assume a rechargeable LIR / ML / VL coin is an equivalent to CR2032.
Do not leave loose coin cells where children can reach them.