Yes, that's exactly what the problem is. The scanner only has 500ms to get a frame scanned and on to the SD card.
So, we need to go in the opposite direction. To make it easier for the scanner to encode on time we need to increase the QP to 11 or 12.
There is another way, we could, in theory...