Definitely a cool idea... My only thought/suggestion, are you doing rendering even if the canvas is outside the current scroll region? I don't know if ThrreJS or your code already account for this scenario and simple leaves it alone.
Aside: Really would be cool to see a Donkey Kong Country style game as a browser-native 3d, not like the pre-rendered 3d on a 2d surface of the original game. Or, for a more 3D effect, similar to Crash Bandicoot games.