What's wrong with debate? Nothing. Calls for expansion to an 8 team playoff is not to minimize the amount of debate. An expansion to 8 teams has the benefit of reducing the power of "THE Committee" in favor of affirming the relevance of the conference championships and conference championship games. The conferences created the CFP, but to actually promote the importance of the conferences (and every team in them), there needs to be expansion. Expansion elevates the importance of metrics over perception. It rewards the eligibility for the CCG and winning of the conference championship, not an easy task, while also rewarding multiple teams worthy of being in a playoff that, for example, lose a divisional tie-breaker and don't qualify for their conference championship despite finishing tied for first in their division (e.g., Alabama 2017, Ohio State 2015). It also virtually eliminates the risk of sectionalism, as all five conference champions guarantee that all regions/fans have a rooting interest in the playoffs, which not only maximizes profits to be shared among the conferences, but also ensures that the most important games in the college football season are watched by the entire country and not one or two regions.
I wouldn't require the top non-P5 get in, only that it have the opportunity. For example, this year it would have gotten to a debate between UCF or Auburn. Isn't it interesting those are the teams the Committee has facing off against each other in a bowl? Some years there might be a UCF or Houston determined worthy while others there might be a 10-2 non P5 who's considered the best of that group, but not really worthy over another non-champion of a P5 conference. The Committee's role would be to select the three at large contestants and to rank the top 8. As the OP suggests, the higher ranked host the lower ranked. Take a week, maybe two week break after the CCGs for final exams, rest up, etc. and then play the games. Followed by the semi-final bowl games as they are currently scheduled to take place.
I think most of the resistance has to do with ESPN being the broadcast partner and they (a) currently and until the CFP agreement runs out have agreed to carry the two semi-final and the championship game, and (b) can't or don't want to commit to carrying an additional 4 quarterfinal games. Everyone talks about the players. What about them. FCS, D2, D3 all play a 16 team playoff (D3 might be a 32). If an extra game is such a big deal, go back to playing 11 games in the regular season. We now know that up-scheduling and playing that tough non-conf foe really doesn't matter. Have all conferences play a mandated 9 game (or 8 game, whatever, just make it standard) season with 2 or 3 NCGs for 11 total, the CCG, and then the quarterfinal game.
Interestingly, there have been a total of 9 teams that have made the playoffs since it started in 2004. Only four have gone multiple times (Bama 4, Clemson 3, OU 2, OSU 2). There's a real concentration of power in the FBS game and I don't think that's healthy for the continued popularity of the game. Not just the recruiting benefits, but also the positive media exposure for the participating schools in terms of more student applicants, etc. is of high value. The conferences are going to mandate expansion to 8 teams. CCGs are expensive to put on and if there's really no benefit, only a downside (for this year Wisconsin is the best example), then those are going to stop and that's not good for the game.