I always assumed that video quality got horrible because of /e/, but it turns out that changing the quality setting in the camera settings from 1080p to 4K leads to massively improved video quality!
Apparently instead of using downsampling to convert 4K to 1080p, the camera app just gets a terrible quality version with excessive oversharpening from the driver if it is set to 1080p.