wiki:NetworkBugUnitTests

Unit Tests for Network Bugs


In order to evaluate NetCheck a suit of traces with known issues and bugs were fed through NetCheck to understand how well NetCheck is able to identify real network bugs.


Unit Tests for Known Network Issues


In order to determine how well NetCheck is able to verify an application, we searched the Internet for various bug reports that have caused applications to break. A list of these bugs is compiled and a set of unit tests are generated, which trigger the bugs that are reported. All unit tests are written in Python as to make the tests portable across various systems. The tests were run in a VM using VirtualBox. The host OS was Windows 7 and the guest OS was Ubuntu 12.04. For the traces that represented Windows and BSD, the unit tests were run on Linux and the important network traces were modified to represent how the trace would have looked on Windows or BSD as was reported or observed on those systems.

A total of 71 traces are generated, for 24 different categories of bugs. Then we passed these traces to NetCheck and the results are shown in Table 1. The unit tests are grouped into several different categories. Each unit test is not meant to generate an error, rather generate different behaviors under different circumstances. As seen in Table 1, NetCheck is able to correctly identify 95.7% of the reported bugs!

Table 1: NetCheck's validation on reported network bugs

Unit Test Category (#Traces) Bugs Detected/Number of Bugs Incorrect/Erroneous Errors Bug Description
bind_local_interface (2) 2/2 0 In these two unit tests, the socket was binded to the local interface (127.0.0.1), then tried to connect to a non-loopback address and failed with an error. The two tests display the variation of the system call connect() on Linux and on Windows.
block_udp_close_socket (2) 1/1 0 In these two tests, the udp socket was blocking on the call recvfrom() when either the close() or shutdown() call was made on the socket from a different thread.close() was called, the socket kept blocking indefinitely on recvfrom(), however when shutdown was called, the recvfrom() call immediately returned an error. 1, 2
block_tcp_close_socket (2) 1/1 0 In these two tests, the tcp socket was blocking on the call recv() when the close() and shutdown() call was made on the socket from a different thread. When close() was called, the socket kept blocking indefinitely on recv(), however when shutdown was called, the recv() call immediately returned an error. 1, 2, 3
broadcast_flag (2) 2/2 0 In these two unit tests, the user tried to send a broadcast message on a socket without setting or using the SO_BROADCAST option and was unsuccessful. The tests show the behavior on Linux and on Windows.
buffer_full (1) 1/1 0 In this unit test, the user tried to send a message on a socket while the send buffer was already full and was unable to do so. This behavior varies between Windows and Unix system. 1
invalid_port (3) 3/3 0 In these tests the client tried to connect to a port that the server was not listening on. In the first test the client socket made a blocking connect() on Linux, in the second test the client socket made a non-blocking connect() on Linux, and in the third test the client socket made a non-blocking connect() on Windows (the behavior observed is different from the second case).
loopback_address (7) 0/0 0 In these unit tests, the calls bind() and getaddrinfo() were made with variations of the loopback address, such as '127.0.0.1', '0.0.0.0' and the empty string. Both Python and OSes, accept all variations of the loopback address without any error. 1, 2, 3
multicast_issue (3) 3/3 0 These unit tests consist of variations on multicast issues, such as setting the IP_ADD_MEMBERSHIP and binding to a non-multicast address. In all cases, the server did not receive the message sent to the multicast address. The variations are shown both in Linux and Windows. 1
multiple_bind (1) 1/1 0 First, bind to the same IP address and port 10 times, then call listen() on the 10th socket that was bound to the (IP, port), followed by binding an 11th socket to the same (IP, port). The first 10 binds were all successful while the 11th bind was unsuccessful after the 10th socket called listen().
nonblock_connect (13) 8/9 1 These 13 unit tests are multiple combinations of non-blocking connect() call followed by other non-blocking connects or non-blocking connect followed immediately by a recv() call. The behavior of the unit tests varied greatly between Linux and Windows as well as the timing of the call. For example a non-blocking connect followed by another non-blocking connect would result in an EINPROGRESS error followed by an ECONNREFUSED or a successful connect. A recv() after a non-blocking connect may result in ECONNREFUSED, EWOULDBLOCK or receiving a message. 1,
nonblock_flag_inheritance (2) 1/1 0 In these two tests we set the server's socket option to non-blocking then called accept() on the socket, and on a BSD machine the accepted socket inherited the non-blocking flag, while on Linux other it did not. 1, 2
oob_data (5) 5/5 0 These unit tests sent multiple Out-Of-Band data part of which were not received by the client in most of these tests.
readline (1) 0/0 0 In this particular test, we casted a socket as a file and called readline() on the file descriptor. We sent the messages 'Hello' followed by the message 'World\n' but the receiver only received 'World\n'. We found that this was a bug in Python, as the system call showed that both of the messages having received. 1
recvtimeo (1) 1/1 0 In this test we set a timeout using the SO_RCVTIMEO option, then immediately called getsockopt() on SO_RCVTIMEO and received a different value from the one we set using setsockopt(). 1, 2
setsockopt_misc (3) 2/2 0 These tests involved setting various socket options that caused network misbehaviors such as setting an option after closing the socket or setting an option that triggers sending a reset packet. NetCheck was able to detect all of these misbehaviors.
shutdown_reset (1) 0/1 1 In this test, the receiving socket called shutdown after receiving a reset packet from the server. The current implementation of NetCheck is not able to differentiate between a reset packet sent explicitly and one that occurred due to network misbehavior.
sigstop_signal (3) 0/0 0 These tests involved sending the SIGSTOP signal to the application while the socket was blocking on some call. In all cases the trace showed the error ERESTARTSYS being returned by the call followed by the call being made again successfully.
so_linger (5) 2/2 0 These tests involved setting the SO_LINGER flag for a socket. On BSD platform, this flag is ignored and the network behavior does not adhere to the expected behavior The two cases were setting SO_LINGER and SO_DONTLINGER.
so_reuseaddr (2) 2/2 0 In these two unit tests, the SO_REUSEADDR flag was set in order to bind multiple sockets to the same (IP, port). The two tests show the difference in behavior of how this flag is handled on Linux and on BSD. In one case we were able to bind a second socket to the same (IP, port) using the SO_REUSEADDR option, in another case we were not able to bind the second socket. 1, 2
tcp_nodelay (3) 0/0 0 In these unit tests, we tried to set the TCP_NODELAY option on an UNIX socket. The error returned is different between Linux, Mac and NetBSD. 1, 2
tcp_set_buf_size_vm (4) 4/4 0 In these tests we used a combination of setting the options SO_SNDBUF and SO_RCVBUF and attempted to send and receive messages bigger than the set values for the buffer sizes. These tests show that we were able to send and receive messages larger then the set buffer size. These behaviors were observed when run in a VM.
udp_large_datagram_vm (2) 2/2 0 In these tests we sent extremely large udp datagrams over the loopback address. We noticed that there was a very high data loss rate (about 70%!) These behaviors were observed when run in a VM.
udp_set_buf_size_vm (2) 2/2 0 In these tests we used a combination of setting the options SO_SNDBUF and SO_RCVBUF and attempted to send and receive datagrams bigger than the set values for the buffer sizes. These tests show that we were able to send and receive datagrams larger then the set buffer size. These behaviors were observed when run in a VM.
vary_udp_datagram (1) 1/1 0 In this test the client sent udp datagrams with varying size: from 32 bytes to 16384 bytes. The server was listening for datagrams of size 1024 only. The server received the entire datagrams if they were smaller than or equal to 1024 bytes. For the larger datagrams, the server received only the first 1024 bytes for each of them.
Total #Traces: 71 44/46 (95.7%) 2 (4.3%)


Last modified 4 years ago Last modified on Sep 26, 2013 4:32:31 PM