Greg Tran from the Harvard Graduate School of design made a fascinating video for his thesis work. It shows how a virtual world could be placed over the physical world, making for example the interior of a building vary in appearance depending on the person entering (visitor, student, or staff member) or the time of day (visual blocks for areas so people can work). The vision he shows is based on a 3D virtual reality that can be manipulated 3D and thus goes beyond watching in your chair with a weird set of glasses on. 
Thinking through how this would work does make it clear that it remains the adult version of an imaginary play. If you do not want to play along you take out the lenses that are used to project the images in your eyes and you can walk through walls perceived by the ones still into the virtual reality. Just as in school playgrounds such party crash behavior will probably not be appreciated, and not before long there will be rules and sanctions on those not willing to play along.
Another issue is how the other senses will be involved (or confused). People are a very visual type of breed but we still listen, smell and touch. Projecting a visual barrier to create privacy does not stop sound, so sharing dirty secrets in a virtually separated area might not be the greatest idea. And you may project something appearing to be paradise, but if you do so on a rubbish dump you would still smell reality. So virtual 3D would not change reality but add another fascinating layer to it.
