I sort of have the same dilemma. I have two core philosophies of choosing where to hunt. The problem is, they are polar opposites. So which to follow? They both have proven productive.
Yeah I know the feeling. I like to actually see deer when I'm deer hunting, but I also like chasing big guys. They're not always synonymous. It's been my experience that old bucks seem to prefer solitude, space away from everybody else. Killing him usually means hunting where the other deer are not, which means a whole lot of inactive boredom for 30 seconds of rush.
I always hear about guys killing big bucks near the parking lot, places ignored or passed up by other hunters. It's places deer wouldn't normally be. I'm a firm believer if you're seeing a bunch of deer, you're probably not going to see a reclusive old fart hanging out with them. Sure you might see one cruise through if there's a hot doe, but where's he at the rest of the season?