I don't know exactly what it is about "felt" cold, but Southern cold is different than up north or out west. I have a brother that lives at 9,500 feet in the mountains above Denver. Their winter temps are usually hanging around zero degrees. He will send me pictures of himself out shoveling snow or sitting on his back deck in short sleeves when it's in the teens, and he's perfectly comfortable. But when he comes hunting in TN, 25 degrees and he's freezing his butt off in the stand. Our cold just somehow "feels" colder. Perhaps it is the moisture in the air.
To add to that, I can hunt here down into the teens and be fairly comfortable. However, the coldest I have ever been afield was fishing in Florida with the temps in the low 40s. I dropped into hypothermia faster than I thought possible. Something about cold, moist air really sucks the heat out of you.