Ghosting & distancing might save you a confrontation now, but you just trade current awkwardness for later awkwardness.

Say something when you want to disengage. It's considerate & it defuses current and future awkwardness.

Even if you don't endorse your feelings, you can still share them & say you're conflicted about them. Better than staying quiet. In fact I expect that sharing gets you 80% of the way to solving whatever interpersonal issues would have come of it.