You laid out every reason why you shouldn't switch to a 20 ga and there's nothing wrong with that. It's a rare occurrence but I still get the urge to carry a 12 ga (3" 2 oz TSS) from time to time.I've considered for years putting together a 20 gauge and just never got around to it, now I've reached the point where I just don't understand the desire.
1. There is no extra credit for hunting with a 20, it's not like a recurve killed deer versus a rifle.
2. My 12 isn't that heavy to tote, and I like the extra weight for fitness reasons.
3. If I miss or potentially wound a bird due to a sub gauge and misjudging yards or just because I'd be furious I didn't have the 12.
4. I shoot TSS out of my 12, a 3.5" 2 1/2 oz load so killing power is exponentially more than a 20 one would think
5. Recoil is unnoticed on a 12
Those are just a few of the random thoughts I have rattling around. No judgement to those who do shoot smaller gauges, I just fail to see the point.
Those of us who have switched to primarily 20 ga did so because we likely have opposing takes to each one of your points. As for #1, I've never looked at it as earning "extra credit" or winning cool points.
All that matters is that you're confident and effective with what you decide to hunt with. It's all personal preference at the end of the day.