Tag: connection

I have a simplest/common network architecture.

Web server sits behind router on local network. This router does iptables DNAT so port forwarding is achieved to web server.

Therefore, I’m able to download file from server 1 to my computer over the internet.

My questions

1. What is the proper kernel tuning to ensure that router is using most of its potential (for around 2000 connections and highest throughput)? I have an issue in ORANGE

2. Do kernel parameters look fine on Server 1?

3. Can you explain why I’ve got just 3mbps from Server 1 while CPU and RAM are not overloaded? So can you see other issues apart Linux kernel, CPU and RAM? Could you list these possible issues to explore? 1gbps network interfaces, ports, etc? 2×1.5ghz ARM is slow for routing? iptables version?

OS and resources

Computer – Mac OS 8 x86 CPU cores, 16G/32G of free RAM

Router – Linux DD-WRT 2 ARM CPU cores, 270M/512M of free RAM

Server 1 – Linux Ubuntu 18.04 4 x86 CPU cores, 240M/32G of free RAM (500M swapped to SSD)

Server 2 – Linux Raspbian 1 ARM CPU core, 95M/512M of free RAM

MTU

Everywhere 1500

TXQUEUELEN

Everywhere 1000

Protocols

UDP speeds are fine

TCP speed is affected, any port

Iptables version

Router – 1.3.7

Server 1 – 1.8.4

Server 2 – 1.6.0

Linux versions

Router – 4.9.207

Server 1 – 5.4.0-67-generic

Server 2 – 4.14.79+

Theoretical link speeds

From my computer to router – 30mbps / 3.75 MB/s

From router to web server 1 – 1gbps

From router to web server 2 – 1gbps

Download speeds from web server (file is hosted in RAM)

TEST 1: Server 2 -> Router = 800mbps

TEST 2: Server 2 -> Computer = 30mbps

TEST 3: Server 1 -> Router = 800mbps

TEST 4: Server 1 -> Computer using 15 connections = 15mbps

TEST 5: Server 1 -> Computer = 3mbps (the issue!)

CPU usage is at around few percents on any device. CPU load average is 0.0x on all devices, but Server 1 – it has 4.6 load average. Server 1 also handles around 500-1000 connections for other things outside of tests, but at around 1mbps so it shouldn’t affect test throughput dramatically (unless these connections somehow making things worse indirectly).

Regardless that load is higher, TEST 3 performed very well. So it’s still hard to blame Server 1.

There are no issues in dmesg on any device.

My thoughts

Issue appears only when DNAT’ing on router and only with Server 1 which has high amount of other connections (but these connections are almost idling so shouldn’t affect everything badly?).

Most interesting test to describe in final thoughts

When I do multi-thread web download (TEST 4) Server 1 performs much better.
So it’s capable to reach higher download speeds. But why 1 connection can’t reach same speed as multiple ones?

Parameters that I explored

Can you see something that is not well optimised for Linux router?

net.core.wmem_max – maximum tcp socket send buffer memory size (in bytes). Increase TCP read/write buffers to enable scaling to a larger window size. Larger windows increase the amount of data to be transferred before an acknowledgement (ACK) is required. This reduces overall latencies and results in increased throughput.

This setting is typically set to a very conservative value of 262,144 bytes. It is recommended this value be set as large as the kernel allows. The value used in here was 4,136,960 bytes. However, 4.x kernels accept values over 16MB.

Router – 180224

Server 1 – 212992

Server 2 – 163840

Somewhere else used – 83886080

net.core.wmem_default

Router – 180224

Server 1 – 212992

Server 2 – 163840

Somewhere else used – 83886080

net.ipv4.rmem_max – maximum tcp socket receive buffer memory size (in bytes)

Router – 180224

Server 1 – 212992

Server 2 – 163840

Somewhere else used – 335544320

net.core.rmem_default

Router – 180224

Server 1 – 212992

Server 2 – 163840

Somewhere else used – 335544320

net.ipv4.tcp_rmem – Contains three values that represent the minimum, default and maximum size of the TCP socket receive buffer. The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent) especially for 10 Gigabit adapters.

Router – 4096 87380 3776288

Server 1 – 4096 131072 6291456

Server 2 – 4096 87380 3515840

Somewhere else used – 4096 87380 4136960 (IBM)

net.ipv4.tcp_wmem – Similar to the net.ipv4.tcp_rmem this parameter consists of 3 values, a minimum, default, and maximum. The recommendation is to use the maximum value of 16M bytes or higher (kernel level dependent) especially for 10 Gigabit adapters.

Router – 4096 16384 3776288

Server 1 – 4096 16384 4194304

Server 2 – 4096 16384 3515840

Somewhere else used – 4096 87380 4136960 (IBM)

net.ipv4.tcp_tw_reuse – In high traffic environments, sockets are created and destroyed at very high rates. This parameter, when set, allows no longer needed and about to be destroyed sockets to be used for new connections. When enabled, this parameter can bypass the allocation and initialization overhead normally associated with socket creation saving CPU cycles, system load and time.

The default value is 0 (off). The recommended value is 1 (on).

Router – 0

Server 1 – 2

Server 2 – 0

Somewhere else used – 1

net.ipv4.tcp_tw_reuse

Router – 0

Server 1 – 2

Server 2 – 0

Somewhere else used – 1

net.ipv4.tcp_max_tw_buckets – Specifies the maximum number of sockets in the “time-wait” state allowed to exist at any time. If the maximum value is exceeded, sockets in the “time-wait” state are immediately destroyed and a warning is displayed. This setting exists to thwart certain types of Denial of Service attacks. Care should be exercised before lowering this value. When changed, its value should be increased, especially when more memory has been added to the system or when the network demands are high and environment is less exposed to external threats.

Router – 2048

Server 1 – 131072

Server 2 – 2048

Somewhere else used – 65536, 262144 (IBM), 45000 (IBM)

net.ipv4.tcp_tw_reuse

Router – 0

Server 1 – 2

Server 2 – 0

Somewhere else used – 1

net.ipv4.tcp_fin_timeout

Router – 60

Server 1 – 60

Server 2 – 60

Somewhere else used – 15

net.ipv4.tcp_max_syn_backlog

Router – 128

Server 1 – 2048

Server 2 – 128

Somewhere else used – 65536

net.ipv4.ip_local_port_range – range of ports used for outgoing TCP connections (useful to change it if you have a lot of outgoing connections from host)

Router – 32768 60999

Server 1 – 32768 60999

Server 2 – 32768 60999

Somewhere else used – 1024 65535

net.core.netdev_max_backlog – number of slots in the receiver’s ring buffer for arriving packets (kernel put packets in this queue if the CPU is not available to process them, for example by application)

Router – 120

Server 1 – 1000

Server 2 – 1000

Somewhere else used – 100000, 1000 (IBM), 25000 (IBM)

net.ipv4.neigh.default.gc_thresh1

Router – 1

Server 1 – 128

Server 2 – 128

Somewhere else used – 128

net.ipv4.neigh.default.gc_thresh2

Router – 512

Server 1 – 512

Server 2 – 512

Somewhere else used – 512

net.ipv4.neigh.default.gc_thresh3

Router – 1024

Server 1 – 1024

Server 2 – 1024

Somewhere else used – 1024

net.ipv4.neigh.default.gc_thresh3

Router – 1024

Server 1 – 1024

Server 2 – 1024

Somewhere else used – 1024

net.core.somaxconn – maximum listen queue size for sockets (useful and often overlooked setting for loadbalancers, webservers and application servers (like unicorn, php-fpm). If all server processes/threads are busy, then incoming client connections are put in “backlog” waiting for being served). Full backlog causes client connections to be immediately rejected, causing client error.

Router – 128

Server 1 – 4096

Server 2 – 128

net.ipv4.tcp_mem – TCP buffer memory usage thresholds for autotuning, in memory pages (1 page = 4kb)

Router – 5529 7375 11058

Server 1 – 381144 508193 762288

Server 2 – 5148 6866 10296

net.nf_conntrack_max – maximum number of connections

Router – 32768

Server 1 – 262144

Server 2 – no information

net.netfilter.nf_conntrack_max – maximum number of connections? If this is correct parameter, then 1560 is not enough

Router – 1560

Server 1 – 262144

Server 2 – no information

/proc/sys/net/ipv4/tcp_congestion_control – Network congestion in data networking (…) is the reduced quality of service that occurs when a network node is carrying more data than it can handle. Typical effects include queueing delay, packet loss or the blocking of new connections. Networks use congestion control and congestion avoidance techniques to try to avoid congestion collapse.1

Router – westwood

Server 1 – cubic

Server 2 – cubic

net.ipv4.tcp_syn_retries – Specifies how many times to try to retransmit the initial SYN packet for an active TCP connection attempt. The current setting is 20, which means that there are 20 retransmission attempts before the connection times out. This can take several minutes, depending on the length of the retransmission attempt.

Router – 6

Server 1 – 6

Server 2 – 6

net.ipv4.tcp_low_latency – The default value is 0 (off). For workloads or environments where latency is a higher priority, the recommended value is 1 (on).

Router – 0

Server 1 – 0

Server 2 – 0

net.ipv4.tcp_limit_output_bytes – Using this parameter, TCP controls small queue limits on per TCP socket basis. TCP tends to increase the data in-flight until loss notifications are received. With aspects of TCP send auto-tuning, large amounts of data might get queued at the device on the local machine, which can adversely impact the latency for other streams. tcp_limit_output_bytes limits the number of bytes on a device to reduce the latency effects caused by a larger queue size.

Router – 262144

Server 1 – 1048576

Server 2 – 262144

Somewhere else used – 262,144 (IBM), 131,072 (IBM)