Protect critical code paths in 3D viewer

I have yet to see a crash from such an issue, but it could clearly
happen, so let's make it safe