The Ninth Commandment: You Shall Not Desire Your Neighbor's Wife

In the latest Avengers film, Captain America travels back in time to before Peggy when Peggy Carter married this other man and he marries her instead. Would it have been a adultery for him to pursue and marry a girl who he knew was married in his own future time, but not at the time he was currently traveling to? Seems like this might be adultery for him, but not for her. Well, it's definitely not adultery for Peggy Carter. But we do, we are in a territory that at least sounds kind of like we might have a violation of the ninth commandment here. You shall not desire your neighbor's wife.