There's many factors which conspire to make you 'love' someone, all playing out on a subconcious level. Do you like their smell, their hair, respect their intelligence, philosophies, breasts, hobbies, attitude, arse etc.
These are a set of semi-rational desires for characteristics your chosen mate will exhibit. You might have 'loved' someone, but because they didn't fit your criteria (such as taste in music) you might never click off as well as you would have. This is because on some level you rationally decided that someone you share your life with should have good taste in music, so you don't have to listen to rubbish all the time.
The same could be said about penis size, it's just another rational quantifier that plays into the 'love package', if you could choose to be in love with someone with a body that makes you happy, or someone who doesn't - what will people choose?
Then, there is the issue that there is not just one person that you could ever love out there, and you have to accept their cocksize because that's the end of that chapter. You can find someone else who fulfills more of your criteria and then love them instead, which is an even better scenario.