Hello,
I wanted to update this thread with a note that I made a pretty major update to MiniVNC. The thing I wanted to focus on in this release was compatibility with clients other than RealVNC. Due to processor limitations, B&W 68000 Macs can only use the older TRLE encoding and still require the RealVNC client, but color Macs now can use Hextile and ZRLE encodings which vastly improve compatibility with other clients. I also added a cursor pseudo-encoding which lets the client draw the cursor locally, and added an option to hide the remote cursor. This improves the perceived responsiveness of the connection. Lastly, I redesigned the UI to use of separate windows which can be shown and hidden independently, and made logging a whole lot more convenient:
Oh yeah, it also uses a preferences file to store settings between runs. It also allows advanced users to change some more obscure parameters using ResEdit!
To try out this release, go here:
https://github.com/marciot/mac-minivnc/releases/tag/v1.1-beta-jan-8-2024
And now for a more technical run down...
I investigated several VNC clients and found two things that were causing MiniVNC to fail with most clients 1) the use of TRLE encoding and 2) the use of indexed colors.
To address these issues, I implemented the Hextile encoding and TrueColor support. This allowed TightVNC, VNCThing, and MacVNC to function. The last VNC client I had access to was Screen Sharing on macOS High Sierra. This one proved to be a bit of a bear. It always failed to get past initial handshaking, so I could never get the connection far even to know what encodings are supported.
Eventually, I found out it always breaks the connection at an early stage of the handshaking and then immediately reconnects. Due to the way MiniVNC handled disconnected clients, I was not restarting fast enough to pick up on the second connect attempt. Once I resolved this, I learned Screen Sharing would only connect if password authentication was enabled. I did not want to implement DES encryption, so I implemented a fake handshake that sends a challenge string and does not check the reply. The upshot of this is that Screen Sharing now asks for a password, but will connect no matter what you type in.
Once I finally was able to get past the handshaking, I learned that Screen Sharing only supported Zlib and ZRLE encodings. I wasn't sure about how successfully older Macs could run ZLib, but I decided to find the smallest implementation of Zlib I could find and try that. I settled on a library called MiniZ and implemented a new ZRLE protocol. It is very similar to the old TRLE encoding but uses larger tiles and compresses the stream with ZLib. Getting that to work meant that I could now connect using Screen Sharing on macOS!