I might let my knuckles drag on the ground for a bit and explain my thought process.
Whether you believe in evolution of not, men and women have a built in primal instinct just like any other creature on earth.
The bare bones of sex is to reproduce though a handful of creatures developed into sex for pleasure or in the case of Bonobo Chimps, to communicate.
Either way we are hard-wired with a species survival instinct.
Okay so here is my thought process, maybe it is a primal wiring in our subconscious that still wants sex but desires to find a different sex mate of perceived stronger genes.
It is not that we don't love the person we are with but the inner primal instinct is craving a different gene pool.
Men and women or same sex couples can walk hand in hand in public, both in love, but their eye still wanders because of the primal search for stronger genes.
(A basic analogy:
Men are attracted to hips/breasts in a woman, that's primal, detecting breeding-ability.
Women may be attracted to fitness/stamina, nurturing/providing, breeding-ability because of nesting/security which is also primal.)
So are we deeply being primal creatures subconsciously?