Fix integer overflow in MallocFrameBuffer()

If MallocFrameBuffer() returns FALSE, frame buffer pointer is left to
NULL. Subsequent writes into that buffer could lead to memory
corruption, or even arbitrary code execution.

Promote integers to uint64_t to avoid integer overflow issue during
frame buffer allocation for very large screen sizes

x11vnc: fix double X_UNLOCK on xrandr events

check_xrandr_event() assumes X_LOCK is taken before it is called, and
currently calls X_UNLOCK on behalf of the caller. But in practice, all
callers assume that the lock is still held after check_xrandr_event()
returns. In particular, this leads to a double-unlock and crash in
check_xevents() on any xrandr event.

It was reported that LZO has security issues in LMS-2014-06-16-1:
Oberhumer LZO (CVE-2014-4607): http://seclists.org/oss-sec/2014/q2/665

This was also reported by Alex Xu as
https://github.com/LibVNC/libvncserver/issues/9.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Initialize padding in SetFormatAndEncodings' rfbSetPixelFormatMsg.

Use CMAKE_CURRENT_*_DIR instead of CMAKE_*_DIR.

This makes the library friendly to use as a git submodule within another
project, and should change nothing when compiled alone.

For example when having a directory structure like "my_project/external/libvnc",
where in libvnc resides a checkout of libvncserver, one can just reference that
directory from the CMakeLists.txt in my_project with

> add_directory ( external/libvnc )

and add vncclient / vncserver in my_project's taret_link_libraries, one can just
hack away without having to manually make / install LibVNCServer whenever
something is changed there.

x11vnc: adjust blackout region coordinates to the clipping region

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Conflicts:
	.gitignore

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

	https://www.berrange.com/~dan/vencrypt.txtSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

A more up-to-date version of the RFB protocol is maintained by TigerVNC:

	http://sourceforge.net/p/tigervnc/code/HEAD/tree/rfbproto/rfbproto.rstSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Since we connected to the client through the repeater, chances are
that we want this server shut down once the client disconnected.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

UltraVNC offers an add-on to connect clients and servers via IDs with
a so-called repeater (e.g. to bridge firewalled clients and servers):

	http://www.uvnc.com/products/uvnc-repeater.html

This example demonstrates how to use that feature with a
LibVNCServer-based server.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

While at it, also ignore the documentation of the RFB protocol best
downloaded manually from

	http://www.realvnc.com/docs/rfbproto.pdfSigned-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

rfbClientSetClientData() allocates a new rfbClientData, but never gets
cleaned up, which causes memory leaks.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

We will change rfbClientCleanup() to free the data.
Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>

This patch implements support in LibVNCClient for framebuffer updates
encoded as H.264 frames. Hardware accelerated decoding is performed
using VA API.

This is experimental support to let the community explore the possibilities
offered by the potential bandwidth and latency reductions that H.264 encoding
allows. This may be particularly useful for use cases such as online gaming,
hosted desktops, hosted set top boxes...

This patch only provides the client side support and is meant to be used
with corresponding server-side support, as provided by an upcoming patch for
qemu ui/vnc module (to view the display of a virtual machine executing under
QEMU).

With this H.264-based encoding, if multiple framebuffer update messages
are generated for a single server framebuffer modification, the H.264
frame data is sent only with the first update message. Subsequent update
framebuffer messages will contain only the coordinates and size of the
additional updated regions.

Instructions/Requirements:
* The patch should be applied on top of the previous patch I submitted with
minor enhancements to the gtkvncviewer application:
http://sourceforge.net/mailarchive/message.php?msg_id=30323804
* Currently only works with libva 1.0: use branch "v1.0-branch" for libva and
intel-driver. Those can be built as follows:
   cd libva
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install
   cd ..
   git clone git://anongit.freedesktop.org/vaapi/intel-driver
   cd intel-driver
   git checkout v1.0-branch
   ./autogen.sh
   make
   sudo make install
Signed-off-by: David Verbeiren <david.verbeiren@intel.com>

Hide "Connecting" dialog in gtkvncviewer once an update is received.

Hide local cusror in gtkvncviewer.

GCC < 4.6 failed to parse the declaration of ws_header_t correctly because
it did not accept anonymous structs and unions. [1]

Work around the bug by adding names to the unions and structs. Ugly, but
works.

[1] http://gcc.gnu.org/bugzilla/show_bug.cgi?id=4784

byteswap.h exists only on glibc, so building libvncserver with websockets
support was not possible in other systems.

Replace the inclusion of byteswap.h and the WS_* definitions with calls to
htobeNN, which should perform the same conversions, be more portable and
avoid the need to check for the platform's endianness.

libresolv is only present on systems which use glibc; platforms such as
FreeBSD have __b64_ntop as part of libc itself.

Improve the detection process and only link against libresolv if it exists
on the system, and remember to reset CMAKE_REQUIRED_LIBRARIES after
performing the necessary tests, since we do not always want to link against
libresolv.

The current definitions were mostly useful to glibc and followed its
feature_test_macros(3) documentation.

However, this means other platforms still had problems when building with
strict compilation flags. _BSD_SOURCE, for example, is only recognized by
glibc, and other platforms sometimes need _XOPEN_SOURCE instead, or even the
removal of some definitions (such as the outdate _POSIX_SOURCE one).

_POSIX_SOURCE also had to be conditionally defined in some places, as what
it enables or disables during compilation varies across systems.

Building with -ansi failed due to some code (as well as system
headers) using non-C89 features. Fix that by adding the usual
_POSIX_SOURCE and _BSD_SOURCE definitions already present in some
other files.

Using C++-style comments when building the code with -ansi does not
work, so be more conservative with the comment style.

build_dir/rfb is not passed as an include directory automatically to
the compiler, so including that file fails.