Good discrimination in the question: normal runtime when used, but loss while idle, points at the controller not being properly off rather than at a failing cell. A worn cell shortens active runtime first, and yours is fine there.
The usual cause is that the controller is being kept awake or repeatedly woken. An Xbox controller does not have a hard off switch. It sleeps, and it listens enough to wake. If it stays paired to a wireless adapter that remains powered, or to a console in a low-power standby state, it can be woken by handshake traffic and go back to sleep repeatedly, and across several days that adds up to exactly the loss you are describing. The same happens if the controller is sitting somewhere it gets nudged: a slight stick movement or a button brush wakes it fully, and a controller in a drawer with something resting on a trigger will drain completely.
So the first tests are behavioural rather than technical. Charge to full, then turn the controller off by holding the Xbox button until it powers down rather than just walking away, and store it where nothing presses any input. Also power down the wireless adapter, or unplug it, and set the console to a full shutdown rather than standby for the test period. Then measure again after three days. If the loss is much smaller, you have your answer and it is a storage and habit matter rather than a fault.
If it still drops heavily with the controller properly off and nothing pressing it, then the cell or the pack is the suspect. A rechargeable pack that holds a charge poorly while idle is a known ageing behaviour, as is a worn NiMH AA set, and replacing the pack or the cells is the fix.
One genuine check worth doing: feel the controller after it has been idle. If it is even slightly warm, it is not asleep, and that confirms the wake theory immediately without waiting three days.
