My fiancee had an emotional affair...?

We have been having a rocky relationship for a while now. About 6 months ago, we had an explosive fight and I left to visit family so that we could cool things off. He took it as "over" and contacted a fling from a years before (who now has a husband and children). She was unhappy in her marriage, and they talked together about getting married, having children, places to live, etc. They talked about their old memories and how they wish they had ended up together. They even started saying "I love you" to one another. Mind you, she was just a fling, and they were "together" only twice, nothing more. This went on for about two weeks. He eventually told me about her at the urge of this other woman. I came back to gather my things, and he helped me pack. We shared a few goodbye drinks, and I slowly came to terms with things. After a few drinks, we started pouring out some thoughts that we had been hiding from one another. We decided to try it out again; he broke his emotional affair off and told me that he was lonely and didn't know why he did that. He has not done such a thing since. Now, 6 months later, I still can't get over his betrayal. We still fight about it sometimes. Should I let it go, or do I still have the right to be upset? I have never felt his regret about it, he only states that "I left" and that he thought things were over. He has been nothing but honest with me, though and answered all the questions I have asked. What do you think?