Idk about downsides except ugliness and annoying cables.
I use IKEA’s LADDA rechargeables, they are Eneloops in disguise and works super, super well. Still using the first ones I bought without any noticeble loss in duration, and they last a long time. Buy the charger they have as well, it’s very good and you’re set with fresh ones always ready.
And two controllers, each with its own battery pack. One I keep fully charged, resting on the charger, and the other is the one I actively use. When the one in use is low on batteries, I swap it with the one on the charger. This way I never find myself out of juice, and don’t have to remember to plug anything in.