If it’s drawing from your memories, then you might notice that people are more like how you wish they are, or how you tend to think of them, than how they really are. Exactly what’s wrong will depend on how the virtual world was constructed–if it’s based on recordings and data, then things like smells and textures are going to be extremely difficult to replicate. No matter how great the simulated reality is, it won’t be perfect there’ll be little things that just aren’t quite right compared to what you remember.
This is probably what made you suspicious in the first place. Being married to a hologram tends to be a little more major than most inconsistencies…