You have read it correctly and your instinct about leftovers is right. This is cosmetic and harmless, and it does clear.
The player number is not a property of the controller. It is a slot the host assigns, and the controller simply displays whichever slot it was given. On a PS5 the slot is tied to the user profile and the order in which controllers have been registered. If controllers have been paired and unpaired over time, or several user profiles exist, slots can remain reserved, and a newly connecting controller takes the lowest free slot rather than slot one. So player 3 means slots one and two are considered occupied by something, even if nothing is physically connected.
That is also why it does not affect anything functional. Games ask the console for connected controllers and get yours; the number is a label.
To clear it, work from the console side rather than the controller. Check which controllers are registered and remove any you no longer own or use, then check your user profiles and sign out of any you are not using. After that, reset the controller with the pin hole on the back, reconnect by cable, and it should take slot one. Doing the reset without clearing the stale registrations first usually just re-assigns the same number, which is why people conclude the reset does not work.
One thing worth knowing: on a PC the player number behaves differently again, because Windows and Steam each assign their own slot, so a controller can show one number on a console and another on a PC, and that is normal rather than a sign of a problem.
If it still comes up as player 3 after clearing registrations and resetting, the simplest remaining option on a console is to remove and re-add your user, but honestly, for a purely cosmetic label I would weigh whether it is worth that.
