An infatuation can also be explained as a crush. You don't know them but something about them attracts them to you, at least from a distance or due to some idea that your mind probably made up. It's good that you don't act upon these feelings, I guess. But at the same time, it's not like you're married. Maybe your mind is trying to tell you to further explore your options? Or maybe the person you love doesn't have the certain qualities that the others do and you find that intriguing? Or, I mean, you say you love this person, but it seems to me that you don't exactly love this person if you find yourself 'infatuated' with others sometimes (and....infatuated is a strong word, that's why I quoted it like that...but maybe you are, I wouldn't know). Yeah, I know you don't act upon it, but usually people can't think about themselves with anyone else but their partner and they can't find themselves even crushing on another. But I don't know. I hope you find the answer your looking for, but in the end, it's what you think, not what others think. Ask yourself the same question you're asking us, because matters of the heart is really complicated for outsiders to answer, because, well, we aren't you. I don't know if that helped. If not, I apologize, and I hope that someone can answer the question wisely and honestly. I've never experienced such an ordeal, but I tried my best. Good luck on your situation! :)
