This is basically how James Cameron shot Avatar. The backgrounds weren't fully rendered, but had enough detail he could get a feel for the final result. Also the actors would be represented in realtime in his field of view as their 12 foot tall counterparts...pretty amazing.