Fixing two more security issues (remote server crash)

Do not accept a scaling factor of zero on PalmVNCSetScaleFactor and SetScale client->server messages. This would cause a division by zero and crash the server.

Check malloc() return value on client->server ClientCutText message. Client can send up to 2**32-1 bytes of text, and such a large allocation is likely to fail in case of high memory pressure. This would in a server crash (write at address 0).

Merge patches from KDE/krfb

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Fix integer overflow in MallocFrameBuffer()

If MallocFrameBuffer() returns FALSE, frame buffer pointer is left to
NULL. Subsequent writes into that buffer could lead to memory
corruption, or even arbitrary code execution.

Promote integers to uint64_t to avoid integer overflow issue during
frame buffer allocation for very large screen sizes

This allows for reinitializations of e. g. sockets in a SHUTDOWN state.
The only state that doesn't make sense to reinitialize are READY states.

Krfb crashes on quit, if any client is connected
due to a rfbClientConnectionGone call missing

check_xrandr_event() assumes X_LOCK is taken before it is called, and
currently calls X_UNLOCK on behalf of the caller. But in practice, all
callers assume that the lock is still held after check_xrandr_event()
returns. In particular, this leads to a double-unlock and crash in
check_xevents() on any xrandr event.

x11vnc: fix double X_UNLOCK on xrandr events

check_xrandr_event() assumes X_LOCK is taken before it is called, and
currently calls X_UNLOCK on behalf of the caller. But in practice, all
callers assume that the lock is still held after check_xrandr_event()
returns. In particular, this leads to a double-unlock and crash in
check_xevents() on any xrandr event.

It was reported that LZO has security issues in LMS-2014-06-16-1:
Oberhumer LZO (CVE-2014-4607): http://seclists.org/oss-sec/2014/q2/665

This was also reported by Alex Xu as
https://github.com/LibVNC/libvncserver/issues/9.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Initialize padding in SetFormatAndEncodings' rfbSetPixelFormatMsg.

Use CMAKE_CURRENT_*_DIR instead of CMAKE_*_DIR.

This makes the library friendly to use as a git submodule within another
project, and should change nothing when compiled alone.

For example when having a directory structure like "my_project/external/libvnc",
where in libvnc resides a checkout of libvncserver, one can just reference that
directory from the CMakeLists.txt in my_project with

> add_directory ( external/libvnc )

and add vncclient / vncserver in my_project's taret_link_libraries, one can just
hack away without having to manually make / install LibVNCServer whenever
something is changed there.

x11vnc: adjust blackout region coordinates to the clipping region

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Conflicts:
	.gitignore

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

	https://www.berrange.com/~dan/vencrypt.txtSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

A more up-to-date version of the RFB protocol is maintained by TigerVNC:

	http://sourceforge.net/p/tigervnc/code/HEAD/tree/rfbproto/rfbproto.rstSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Since we connected to the client through the repeater, chances are
that we want this server shut down once the client disconnected.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

UltraVNC offers an add-on to connect clients and servers via IDs with
a so-called repeater (e.g. to bridge firewalled clients and servers):

	http://www.uvnc.com/products/uvnc-repeater.html

This example demonstrates how to use that feature with a
LibVNCServer-based server.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

While at it, also ignore the documentation of the RFB protocol best
downloaded manually from

	http://www.realvnc.com/docs/rfbproto.pdfSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

rfbClientSetClientData() allocates a new rfbClientData, but never gets
cleaned up, which causes memory leaks.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We will change rfbClientCleanup() to free the data.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This patch implements support in LibVNCClient for framebuffer updates
encoded as H.264 frames. Hardware accelerated decoding is performed
using VA API.

This is experimental support to let the community explore the possibilities
offered by the potential bandwidth and latency reductions that H.264 encoding
allows. This may be particularly useful for use cases such as online gaming,
hosted desktops, hosted set top boxes...

This patch only provides the client side support and is meant to be used
with corresponding server-side support, as provided by an upcoming patch for
qemu ui/vnc module (to view the display of a virtual machine executing under
QEMU).

With this H.264-based encoding, if multiple framebuffer update messages
are generated for a single server framebuffer modification, the H.264
frame data is sent only with the first update message. Subsequent update
framebuffer messages will contain only the coordinates and size of the
additional updated regions.

Instructions/Requirements:
* The patch should be applied on top of the previous patch I submitted with
minor enhancements to the gtkvncviewer application:
http://sourceforge.net/mailarchive/message.php?msg_id=30323804
* Currently only works with libva 1.0: use branch "v1.0-branch" for libva and
intel-driver. Those can be built as follows:
   cd libva
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install
   cd ..
   git clone git://anongit.freedesktop.org/vaapi/intel-driver
   cd intel-driver
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install
Signed-off-by: David Verbeiren <david.verbeiren@intel.com>