• Christian Beier's avatar
    Fix deadlock in threaded mode when using nested rfbClientIteratorNext() calls. · 3df7537a
    Christian Beier authored
    Lengthy explanation follows...
    
    First, the scenario before this patch:
    
    We have three clients 1,2,3 connected. The main thread loops through
    them using rfbClientIteratorNext() (loop L1) and is currently at
    client 2 i.e. client 2's cl_2->refCount is 1. At this point we need to
    loop again through the clients, with cl_2->refCount == 1, i.e. do a
    loop L2 nested within loop L1.
    
    BUT: Now client 2 disconnects, it's clientInput thread terminates its
    clientOutput thread and calls rfbClientConnectionGone(). This LOCKs
    clientListMutex and WAITs for cl_2->refCount to become 0. This means
    this thread waits for the main thread to release cl_2. Waiting, with
    clientListMutex LOCKed!
    
    Meanwhile, the main thread is about to begin the inner
    rfbClientIteratorNext() loop L2. The first call to rfbClientIteratorNext()
    LOCKs clientListMutex. BAAM. This mutex is locked by cl2's clientInput
    thread and is only released when cl_2->refCount becomes 0. The main thread
    would decrement cl_2->refCount when it would continue with loop L1. But
    it's waiting for cl2's clientInput thread to release clientListMutex. Which
    never happens since this one's waiting for the main thread to decrement
    cl_2->refCount. DEADLOCK.
    
    Now, situation with this patch:
    
    Same as above, but when client 2 disconnects it's clientInput thread
    rfbClientConnectionGone(). This again LOCKs clientListMutex, removes cl_2
    from the linked list and UNLOCKS clientListMutex. The WAIT for
    cl_2->refCount to become 0 is _after_ that. Waiting, with
    clientListMutex UNLOCKed!
    
    Therefore, the main thread can continue, do the inner loop L2 (now only
    looping through 1,3 - 2 was removed from the linked list) and continue with
    loop L1, finally decrementing cl_2->refCount, allowing cl2's clientInput
    thread to continue and terminate. The resources held by cl2 are not free()'d
    by rfbClientConnectionGone until cl2->refCount becomes 0, i.e. loop L1 has
    released cl2.
    3df7537a
Name
Last commit
Last update
VisualNaCro Loading commit data...
classes Loading commit data...
client_examples Loading commit data...
common Loading commit data...
examples Loading commit data...
libvncclient Loading commit data...
libvncserver Loading commit data...
rfb Loading commit data...
test Loading commit data...
utils Loading commit data...
vncterm Loading commit data...
x11vnc Loading commit data...
.gitignore Loading commit data...
AUTHORS Loading commit data...
CMakeLists.txt Loading commit data...
COPYING Loading commit data...
ChangeLog Loading commit data...
Doxyfile Loading commit data...
INSTALL Loading commit data...
LibVNCServer.spec.in Loading commit data...
Makefile.am Loading commit data...
NEWS Loading commit data...
README Loading commit data...
TODO Loading commit data...
acinclude.m4 Loading commit data...
autogen.sh Loading commit data...
configure.ac Loading commit data...
libvncclient.pc.in Loading commit data...
libvncserver-config.in Loading commit data...
libvncserver.pc.in Loading commit data...
ltmain.sh Loading commit data...
prepare_x11vnc_dist.sh Loading commit data...
tightvnc-1.3dev5-vncviewer-alpha-cursor.patch Loading commit data...