Connect to a Mac Remote Desktop using VNC

This will be a quick howto as it’s mostly a settings issue, but here goes:

Macs have come with a Remote Desktop server for quite some time now, and it’s great for using macs to manage macs remotely, though maybe not as nice as an NX or XMing solution.

When trying to manage an Apple computer using a Windows or Linux computer it’s a different story. When you attempt to open the connection the authorization works, but the window will close very quickly, with no apparent error.

The problem lies in the actual implementation of VNC in Apple’s Remote Desktop server (not to be confused with RDP – it’s MUCH slower). Apple has decided to only support one type of tiling, whereas most VNC clients will attempt to find the best solution in order to connect. Specifically, Apple uses HexTile, and if you specify this in the options or properties of your connection, it will work with no problems whatsoever.

If you’d like to make a .VNC configuration file in order to connect to your Mac server using a Windows VNC client (RealVNC used here), just take the following code and save it as a *.VNC configuration file, being careful to change the host from (null) to the remote Apple Remote Desktop server’s IP address (for example: 10.10.10.10).

[Connection]
Host=(null)
[Options]
UseLocalCursor=1
UseDesktopResize=1
FullScreen=0
FullColour=1
LowColourLevel=1
PreferredEncoding=hextile
AutoSelect=0
Shared=0
SendPtrEvents=1
SendKeyEvents=1
SendCutText=1
AcceptCutText=1
DisableWinKeys=1
Emulate3=0
PointerEventInterval=0
Monitor=
MenuKey=F8
AutoReconnect=1

I have tested this method on many Windows and Linux machines, using RealVNC, TightVNC, UltraVNC and even Chicken of the VNC for Mac OS X. It works fine, though I’d like to pound home again that I would really like to have the option to either tunnel application over SSH, or have some type of locally-accelerated RDP-compliant protocol (heck why not use LTSP 5.0?)
One can only dream…

Join the Conversation

21 Comments

  1. I get a “No matching Security Types” error when I try to connect to my OSX Server with this. Have you seen this before? Im using RealVNC.

  2. couldn’t agree more, regarding the note about built-in SSH.

    i’ve set up my Mac Pro to run sshd, then i just use plink & vnc from my xp box for the SSH tunnel. works great.

  3. I also get the “No matching Security Types” error while using Real VNC. It would appear to be a common problem (if the Google results are to be believed) and suggests that people who are able to connect are actually using OSXvnc rather than the built in server.

  4. I am trying to connect to an apple file server using this method… I get an incompatible version error on connect. Any ideas as to a solution? I have been told that I am able to use real vnc to connect to the server. Thanks in advance.

  5. My experience shows that there is a significant difference between the ARD server implementation of OSX and OSX Server.
    The above mentioned method seems to WORK PERFECTLY when I connect to an OSX desktop system (tried with 10.4.x only) regardless of that system running the built-in ARD server or OSXvnc.
    However, the connection NEVER WORKS when I connect to a system running the OSX SERVER edition of Apple’s operating system.

  6. Just to correct my last statement.
    The connection NEVER WORKS when I connect to a system running the OSX SERVER edition of Apple’s operating system using the built-in ARD server. Running OSXvnc on the OSX SERVER does give me access to the server and works well.

  7. Has anyone got this to work? I’m using RealVNC as well and I’ve created the file, changed to the hostname and get exactly the same error as everyone else.

  8. Hi

    The solution to the problem is to tick the VNC Viewers may control screen with password and set a password (found under Computer Settings from Remote Management from within Sharing).

    What I’ve found is that RealVNC performs well, but the screen refresh is flakey – I needed to stop the dock auto-displaying, and it seemed flaky on starting up apps – it does not like the dashboard. TightVNC is a lot slower displaying but seems reliable.

    I haven’t spent too long using either, but I haven’t had to full screen refresh the screen, unlike RealVNC. I am not sure if there is a solution.

  9. Dude, thank you! I know absolutely nothing about Mac’s, but just got one at work for design purposes and will attempt to use the thing full time, but needed a way to RD from home from a Vista Laptop. Simply changing the format and encoding options to hextile (within UltraVNC) allowed me to connect perfectly. Thanks for the tip!

  10. The two key settings that I found the essential settings for the RealVNC viewer 4.4 to be:

    FullColour=1
    PreferredEncoding=hextile

    (Unfortunately, it has some trouble “remembering” these settings, so I have to set them each time, since I don’t want the settings to apply to all my connections).

  11. wrlee, you are so right. Setting the preferredEncoding without setting fullcolor to 1 does not work. thanks for writing this

    alin iacob.

  12. After installing VNC and setting up remote login on MAC X. I finally was able to connect but I have a black screen. I double checked my setting on my XP pro VNC and MAC remote login setting. Everything looks ok but I still get a black screen. I know I am connecting to the MAC because it request my credentials before I connect. Does anyone have the same problem and a solution?

  13. I was still having problems and found changing VNC options to full color solved my problem.

  14. This worked for me.. so thanks very much for the help! (need to make sure full colours is selected as well)

    Cheers 🙂

  15. Even though this post is old, it’s still applicable. I’ve had difficulty establishing (and retaining) a VNC connection from Windows to MacOSX (non-server) for a while now. I haven’t been able to find a suitable configuration. Using RealVNC with the settings you provided works quite well. Thank you.

Leave a comment

Leave a Reply to Mike Cancel reply

Your email address will not be published. Required fields are marked *