leandromoreira /linux-network-performance-parameters
Learn where some of the network sysctl variables fit into the Linux/Kernel network flow. Translations: 🇷🇺
★ 5,832⑂ 531👁 171⑃ 14● 9▤ 0
More from this author
-1
Daily
5,832
stars
531
forks
171
watchers
14
prs
9
issues
0
releases
README
English | Русский | 中文
TOC
Introduction
Linux network queues overview
Fitting the sysctl variables into the Linux network flow
Ingress - they're coming
Egress - they're leaving
How to check - perf
What, Why and How - network and sysctl parameters
Ring Buffer - rx,tx
Interrupt Coalescence (IC) - rx-usecs, tx-usecs, rx-frames, tx-frames (hardware IRQ)
Interrupt Coalescing (soft IRQ) and Ingress QDisc
Egress QDisc - txqueuelen and default_qdisc
TCP Read and Write Buffers/Queues
Honorable mentions - TCP FSM and congestion algorithm
Network tools
References
Introduction
Sometimes people are looking for sysctl cargo cult values that bring high throughput and low latency with no trade-offs and work in every situation. That's not realistic, although we can say that the newer kernel versions are very well tuned by default. In fact, you might hurt performance if you mess with the defaults.
This brief tutorial shows where some of the most used and quoted sysctl/network parameters are located into the Linux network flow, it was heavily inspired by the illustrated guide to the Linux networking stack and many of Marek Majkowski's posts.
Feel free to send corrections and suggestions! :)
Linux network packets flow to memory overview
Fitting the sysctl variables into the Linux network flow
Ingress - they're coming
1. Packets arrive at the NIC
2. NIC will verify MAC (if not on promiscuous mode) and FCS and decide to drop or to continue
3. NIC will DMA packets to RAM, in a region previously prepared (mapped) by the driver
4. NIC will enqueue references to the packets at receive ring buffer queue rx until rx-usecs timeout or rx-frames
5. NIC will raise a hard IRQ
6. CPU will run the IRQ handler that runs the driver's code
7. 7. The driver will schedule NAPI, clear the hard IRQ, and return
8. The driver raises a soft IRQ (NET_RX_SOFTIRQ)
9. NAPI will poll data from the receive ring buffer until netdev_budget_usecs timeout or netdev_budget and dev_weight packets
10. Linux will also allocate memory to sk_buff
11. Linux fills the metadata: protocol, interface, setmacheader, removes ethernet
12. Linux will pass the skb to the kernel stack (netif_receive_skb)
13. It will set the network header, clone skb to taps (i.e. tcpdump) and pass it to tc ingress
14. Packets are handled to a qdisc sized netdev_max_backlog with its algorithm defined by default_qdisc
15. It calls ip_rcv and packets are handed to IP
16. It calls netfilter (PREROUTING)
17. It looks at the routing table, if forwarding or local
18. If it's local it calls netfilter (LOCAL_IN)
19. It calls the L4 protocol (for instance tcp_v4_rcv)
20. It finds the right socket
21. It goes to the tcp finite state machine
22. Enqueue the packet to the receive buffer and sized as tcp_rmem rules
1. If tcp_moderate_rcvbuf is enabled, the kernel will auto-tune the receive buffer
23. Kernel will signalize that there is data available to apps (epoll or any polling system)
24