I think my initial diagnosis was incorrect, the scrambling is happening at 4, 16, and 256. I attempted to record the monitor changes to better illustrate what I'm seeing.
Here is the Toby card without the RAM expansion, only supports 16 colors:
Here is the Toby card with the RAM expansion...