wiki:WebrickTests

Modifying Network Configuration to Produce Bugs with WEBrick


WEBrick is an HTTP server library for Ruby that is widely used, especially in development of prototype Ruby on Rails applications. To test NetCheck’s effectiveness at diagnosing bugs in WEBrick, we reproduced bugs derived from our own experiences and from the Ruby bug tracker and collected execution traces from the affected systems, which we ran through NetCheck. The results are reported in Table 2, which shows that NetCheck is able to identify the main issue in 16 of the traces (out of 20)!

Table 2: NetCheck's Classification of WEBrick bugs

Unit Tests Type of Error Detect Main Issue Extra Issues Incorrect Issues Trace(s)
bind6 Network yes 0 0 client server
bind6-2 Network yes 0 0 client server
clientpermission Application yes 1 0 client server
closethread Network no 2 0 client server
conn0 Network yes 0 0 client server
firewall Network yes 0 1 client server
gethostbyaddr Network no 0 0 client server
httpprox1 Network yes 0 0 client server
httpprox2 Network yes 0 0 client server
keepalive Application yes 0 0 client server
local Network yes 2 0 client server
max1 Application yes 2 0 client1 client2 server
maxconn Application yes 3 3 client server
maxconn2 Application yes 5 0 client server
mtu Network yes 1 0 client server
multibyte Application yes 1 0 client server
noread Application yes 0 0 client server
permission Application yes 0 0 client server
portfwd Network yes 0 0 client server
special Application yes 0 0 client server
Total: 18/20 (90%) 17 4

Details of WEBrick bugs

Unless otherwise specified, all hosts are running Ubuntu Linux (either 11.04, 11.10, or 12.04). The source code for the basic WEBrick HTTP server is:

#!/usr/bin/ruby
require 'webrick'
include WEBrick


s = HTTPServer.new(
 :BindAddress => '0.0.0.0',
 :Port => 3005,
 :DocumentRoot => Dir::pwd
)
trap("INT"){ s.shutdown }
s.start

bind6

Background Information:

This bug (and its dual, bind6-2), are meant to reflect the varying semantics of binding a socket to the :: address. On recent Microsoft Windows operating systems, binding a socket to :: only binds to the IPv6 addresses. On Linux, a socket bound to :: will bind to both IPv6 and IPv4 addresses - unless /proc/sys/net/ipv6/bindv6only is set to 1, in which case a socket bound to :: will only listen for IPv6 requests. Debian defaults to bindv6only = 1.

This issue can cause program failures (if a developer expects that a program bound to :: will listen on IPv4 addresses, and it doesn't) including security issues (a developer expects that a program bound to :: is only listening on IPv6 addresses, and does not take any steps to secure IPv4 addresses).

NB: NetCheck assumes the Linux behavior (a socket bound to :: is bound to IPv6 and IPv4 addresses), so if a host is listening on ::, NetCheck should expect connections to that hosts' IPv4 address to succeed. However, it will print a warning about the non-portability of :: in any case.

In this trace, the observed behavior is the Windows / Debian behavior (a socket bound to :: is only bound to IPv6 addresses).

References:

Trace information:

  • On the server, run the following:
    sysctl -w net.ipv6.bindv6only=1
    
  • Server is at IPv4 address 128.238.66.220, and is running a WEBrick server that is listening on :: (port 3005)
  • Client is at 128.238.255.159, and uses wget to download a file from the WEBrick server at 128.238.66.220:3005 (using the server's IPv4 address)

bind6-2

Background Information:

See bind6 for more information.

In this trace, the observed behavior is the default Linux behavior (a socket bound to :: is bound to IPv6 and IPv4 addresses).

Trace information:

  • On the server, run the following:
    sysctl -w net.ipv6.bindv6only=0
    
  • Server is at IPv4 address 128.238.66.220, and is running a WEBrick server that is listening on :: (port 3005)
  • Client is at 128.238.38.66, and uses wget to download a file from the WEBrick server at 128.238.66.220:3005 (using the server's IPv4 address)

clientpermission

Background Information:

In this trace, the client connects to a WEBrick server to download a file, but does not have permission to write the downloaded file to the disk. This is obviously an application-layer issue.

Trace information:

  • On the client, run the following (note that this will not work on NTFS file systems):
    touch file.txt
    chmod a-w file.txt
    
  • Server is at IPv4 address 128.238.66.220, and is running a WEBrick server that is listening on port 3005
  • Client is at 128.238.38.67, and uses wget to download a file from the WEBrick server at 128.238.66.220:3005

closethread

Background Information:

If a blocking call is in progress on a socket in one thread and another thread closes the same socket, will the thread in the blocking call know this and raise an exception, or will it continue to block forever? The behavior of the socket API in this situation varies across different implementations.

References:

Trace Information:

  • WEBrick server is listening on 128.238.66.220:3005
  • The client runs the following Ruby script:
    #!/usr/bin/ruby
    
    require 'socket'
    require 'thread'
    
    host = 'omfserver-witest.poly.edu'
    port = 3005
    
    @sock = TCPSocket.open host, port
    
    x = Thread.new do
      sleep(5)
      @sock.close
      puts "Closed socket"
    end
    
    @sock = TCPSocket.open host, port
    
    @sock.print "GET /file.txt HTTP/1.1\r\n"
    @sock.puts "\r\n"
    while line = @sock.readline
        #puts line.chop
    end
      
    @sock.close
    

conn0

Background Information:

The semantics of the address 0.0.0.0 varies across platforms. On Mac / Linux platforms, packets addressed to 0.0.0.0 are routed to the loopback interface (127.0.0.1). On Windows hosts, the address 0.0.0.0 is not valid as a destination address, and raises an error.

References:

Trace Information:

  • Server is at IPv4 address 128.238.66.220, and is running a WEBrick server that is listening on 0.0.0.0, port 3005
  • Client runs the following script (note that we must use a custom script rather than a tool like wget. wget will throw an error at the address 0.0.0.0, but it is an application-layer error, not from the network stack):
    #!/usr/bin/ruby1.9.1
    
    require 'socket'
    require 'thread'
    
    host = '0.0.0.0'
    port = 3005
    
    @sock = TCPSocket.open host, port
    @sock.print "GET /index.html\r\n"
    #@sock.print "GET /index.html HTTP/1.1\r\n"
    @sock.print "\r\n"
    
    while line = @sock.readline
        puts line
        break if @sock.eof?
    end
    
    @sock.shutdown
    

firewall

Background Information:

In this experiment, the server is configured with iptables rules to drop incoming traffic from the client. Note that the results would be different if the firewall was configured to reject traffic rather than drop traffic.

Trace Information:

  • The server is at IP address 128.238.66.220, and is listening with WEBrick server on port 3009. The following iptables rules were set on the server:
    iptables -I INPUT -p tcp --destination-port 3009 -s 128.238.255.159 -j DROP
    
  • The client is at 128.238.255.159 and uses wget to download a file from the WEBrick server at 128.238.66.220:3009.

gethostbyaddr

Background Information:

This experiment shows a limitation of NetCheck with regard to socket API calls that are not native system calls. In this experiment, a gethostbyaddr call hangs for some time and then returns NULL.

However, gethostbyaddr it is not a native system call, it calls a library routine (e.g., avahi on Ubunutu Linux). NetCheck only models native system calls, so it cannot detect this issue.

httpprox1

Background Information:

In this experiment, the client is configured to use an HTTP proxy instead of accessing the WEBrick server directly. From the network traffic point of view, this looks similar to a client using a NAT.

Trace Information:

  • The server is at IP address 128.238.38.71, and is listening with WEBrick server on port 3005.
  • The client is at configured to use a proxy for HTTP connections.
  • The HTTP proxy is running on port and has address 10.0.0.200 on the network facing the client, and 128.238.66.220 on network facing the server.

httpprox2

Background Information:

In this experiment, the client is again configured to use an HTTP proxy instead of accessing the WEBrick server directly. This time, the HTTP proxy already has a cached copy of the document, so it does not contact the WEBrick server at all. From the network traffic point of view, this looks similar to the case where the "wrong" server responds to a request (local).

Trace Information:

  • The server is at IP address 128.238.38.71, and is listening with WEBrick server on port 3005.
  • The client is at configured to use a proxy for HTTP connections.
  • The HTTP proxy is running on port and has address 10.0.0.200 on the network facing the client, and 128.238.66.220 on network facing the server.

keepalive

Background Information:

Some things appear to be network issues because the developer does not fully understand a platform or protocol - for example, differences in behavior between HTTP 1.0 and HTTP 1.1. NetCheck should not (and does not) flag these as network errors.

Trace Information:

  • The server is listening with WEBrick server on port 3005.
  • Both server and client are on the same host, so that loopback interface address (127.0.0.1) is used for both.
  • The client runs the following code:
    #!/usr/bin/ruby1.9.1
    
    require 'socket'
    require 'thread'
    
    host = '127.0.0.1'
    port = 3005
    
    @sock = TCPSocket.open host, port
    #@sock.print "GET /index.html\r\n"
    @sock.print "GET /index.html HTTP/1.1\r\n"
    @sock.print "\r\n"
    
    while line = @sock.readline
        puts line
        break if @sock.eof?
    end
    
    @sock.shutdown
    

local

Background Information:

In this experiment, the client is misconfigured so that the hostname of the server on which WEBrick is hosted resolves to another address (in this case, the local 127.0.0.1 address). The client's HTTP request is answered - but not by the server it expects.

This is a network error, and is classified as such by NetCheck.

max1

Background Information:

In this experiment, a WEBrick server is configured to only serve a single HTTP request at a time. Subsequent requests are not served until the first is completed, and may time out while waiting, but this should not raise an error.

Trace Information:

  • The server is at 128.238.38.71 and is running a WEBrick server listening on port 3005.
  • The WEBrick server is configured to only serve a single request at a time:
    #!/usr/bin/ruby
    require 'webrick'
    include WEBrick
    s = HTTPServer.new(
     :MaxClients => 1,
     :Port => 3005,
     :DocumentRoot => Dir::pwd
    )
    trap("INT"){ s.shutdown }
    s.start
    
  • A client at 128.238.38.67 connects to the server and starts downloading a large (10MB) file.
  • While this first download is going on, a second client at 128.238.66.220 connects to the server and requests the same file. The client is configured to time out and give up if the server does not respond within 2 seconds:
    wget -qO- --timeout=2 --tries=1 http://128.238.38.71:3005/index.html
    

maxconn

Background Information:

In this trial, a client used the Apache benchmark tool to make multiple concurrent connections to a WEBrick HTTP server. Although this was done under 'normal' network conditions, NetCheck indicates (incorrectly) that a network error was present.

This reveals a limitation of NetCheck: when there are multiple non-blocking 'connect' calls from a single client to a server, NetCheck cannot ordinarily match client-side sockets to its corresponding server-side socket, and may draw wrong conclusions as a result.

This limitation may be overcome by modifying the source code of the application so that the client calls getsockname() on every socket that connects to the server. Then both the client trace and the server trace includes the client port number, which makes connection matching trivial.

maxconn2

Background Information:

This trace is equivalent to the maxconn case, but in maxconn2 a large file was requested instead of a small index.html.

mtu

Background Information:

This trace was gathered on a network that suffers from the MTU black hole problem. Small packets (such as those in the TCP handshake or the HTTP GET request) will traverse the network successfully, but large packets (such as those in the body of the HTTP response) will be dropped by the network. This issue (partial data loss) is detected by NetCheck and classified as a network problem.

The trace has a second network-related issue: the client in this trace is behind a NAT.

multibyte

Background Information:

On some versions of Ruby, WEBrick uses size to calculate the content-length in the HTTP header. It gives the wrong value for responses with multibyte characters, and the result is that the client does not download the entire file that is requested.

References:

Trace Information:

  • To replicate this bug, we reverted the changes described here on the Ruby installation on the server.
  • The server ran the following WEBrick script:
    #!/usr/bin/ruby1.9.1
    # encoding: utf-8
    
    
    require 'webrick'
    
    include WEBrick
    
    hello_proc = lambda do |req, resp|
     resp['Content-Type'] = "text/html"
     resp.body = %{
       <html>
         <head>
           <title> Привет, Rails! </title>
         </head>
         <body>
          <h1> Привет, Rails! </h1>
          <p>
            Сейчас Mon Jan 28 14:14:13 +0200 2008
          </p>
          <p>
            Настало время
            <a href="/say/goodbye">попрощаться!</a>
          </p>
         </body>
       </html>
     }
    end
    
    hello = HTTPServlet::ProcHandler.new(hello_proc)
    
    s = HTTPServer.new(:Port => 3000)
    s.mount("/", hello)
    
    trap("INT"){ s.shutdown }
    s.start
    
  • The client ran
    wget -qO- http://localhost:3000/
    
  • Both were on the same host, so the client and server IP address is 127.0.0.1

noread

Background Information:

In this trace, the client makes a TCP connection to a WEBrick server and sends an HTTP GET request, but then shuts down without calling read() on the socket. This is a logical application error.

Trace Information:

  • The server is on 128.238.38.71 and is running a WEBrick server that listens on port 3005
  • The client is at IP 128.238.38.67 and runs the following script:
    #!/usr/bin/ruby
    
    require 'socket'
    require 'thread'
    
    host = '128.238.38.71'
    port = 3005
    
    @sock = TCPSocket.open host, port
    @sock.print "GET / HTTP/1.0\r\n"
    @sock.shutdown
    

permission

Background Information:

In this trace, the client connects to a WEBrick server to download a file, but the WEBrick process does not have permission to read the requested file and crashes. This is obviously an application-layer issue.

Trace information:

  • On the server, run the following (note that this will not work on NTFS file systems):
    touch file.txt
    chmod a-r file.txt
    
  • Server is at IPv4 address 128.238.66.220, and is running a WEBrick server that is listening on port 3005
  • Client is at 128.238.255.159, and uses wget to download a file from the WEBrick server at 128.238.66.220:3005

portfwd

Background Information:

The server in this experiment is behind a router that does NAT and port forwarding. If the IP and port information are provided in the config file, NetCheck is still able to process the trace.

Trace Information:

  • The server has public IP 96.250.21.150 and private IP 192.168.1.50. It is running a WEBrick server listening on port 3005 locally, which is mapped to port 3001 on the router.
  • The client is at 128.238.66.220 and runs
    wget -qO- http://96.250.21.150:3001/index.html
    

special

Background Information:

On some versions of Ruby, WEBrick fails if it receives a request with a URI that contains certain characters (such as the "|" character). Obviously, this is a logical application error and not a network error, but it can masquerade as a network error.

References:

Trace Information:

  • The client ran the following script:
    #!/usr/bin/ruby1.9.1
    
    require 'socket'
    
    host = '127.0.0.1'
    port = 3005
    
    @sock = TCPSocket.open host, port
    @sock.print "GET /test&&test.txt HTTP/1.0\r\n"
    @sock.print "\r\n"
    
    while line = @sock.readline
        puts line
        break if @sock.eof?
    end
    
    @sock.print "GET /test|test.txt HTTP/1.0\r\n"
    @sock.print "\r\n"
    
    while line = @sock.readline
        puts line
        break if @sock.eof?
    end
    
    @sock.shutdown
    
Last modified 4 years ago Last modified on Sep 22, 2013 10:17:20 PM