Traffic Control
Contents |
Introduction
Linux's traffic control functionality offers a lot of capabilities related to influencing the rate of flow, as well as latency, of primarily outgoing but also in some cases incoming network traffic. It is designed to be a "construction kit" rather than a turn-key system, where complex network traffic policing and shaping decisions can be made using a variety of algorithms. The Linux traffic control code is also often used by academia for research purposes, where is it can be a useful mechanism to simulate and explore the impact of a variety of different network behaviors. See netem for an example of a simulation framework that can be used for this purpose.
Of course, Linux traffic control can also be extremely useful in an IT context, and this document is intended to focus on the practical, useful applications of Linux traffic control, where these capabilities can be applied to solve problems that are often experienced on modern networks.
Incoming and Outgoing Traffic
One common use of Linux traffic control is to configure a Linux system as a Linux router or bridge, so that the Linux system sits between two networks, or between the "inside" of the network and the real router, so that it can shape traffic going to local machines as well as out to the Internet. This provides a mechanism to prioritize, shape and police both incoming (from the Internet) and outgoing (from local machines) network traffic. The simplest configuration using this approach is to create a bridge device with brctl, add two Ethernet ports to this bridge (again using brctl), and then apply prioritization, shaping and policing rules to both interfaces. One physical interface will be connected to an upstream router on the same network, while the other network port will be connected to a layer 2 access switch to which local machines are connected. This allows powerful egress shaping policies to be created on both interfaces, to control the flows in and out of the network.
Recommended Resources
Resources you should take a look at, in order:
- HTB documentation by Martin Devera
- Differentiated Services On Linux HOWTO by Leonardo Balliache
- A Practical Guide to Linux Traffic Control by Jason Boxman
- IFB - replacement for Linux IMQ, with examples
- Use of iptables hashlimit
Related Interesting Links:
- Second Life Bandwidth Testing Protocol (using Netem)
- UDP Buffer Sizing, part of Topics in High Performance Messaging
Recommended Approaches
Daniel Robbins has had very good results with the HTB queuing discipline - it has very good features, and also has very good documentation, which is just as important, and is designed to deliver useful results in a production environment. And it works. If you use traffic control under Funtoo Linux, please use the HTB queuing discipline as the root queuing discipline because you will get good results in very little time. Avoid using any other queuing discipline under Funtoo Linux as the root queuing discipline on any interface. If you are creating a tree of classes and qdiscs, HTB should be at the top, and you should avoid hanging classes under any other qdisc unless you have plenty of time to experiment and verify that your QoS rules are working as expected. Please see State of the Code for more info on what Daniel Robbins considers to be the current state of the traffic control implementation in Linux.
State of the Code
If you are using enterprise kernels, especially any RHEL5-based kernels, you must be aware that the traffic control code in these kernels is about 5 years old and contains many significant bugs. In general, it is possible to avoid these bugs by using HTB as your root queueing discipline and testing things carefully to ensure that you are getting the proper behavior. The prio queueing discipline is known to not work reliably. See Broken Traffic Control for more information on known bugs with older kernels.
If you are using a more modern kernel, Linux traffic control should be fairly robust.
Inspect Your Rules
If you are implementing Linux traffic control, you should be running these commands frequently to monitor the behavior of your queuing discipline. Replace $wanif with the actual network interface name.
tc -s qdisc ls dev $wanif tc -s class ls dev $wanif
Matching
Here are some examples you can use as the basis for your own filters/classifiers:
- protocol arp u32 match u32 0 0 - match ARP packets
- protocol ip u32 match ip protocol 0x11 0xff - match UDP packets
- protocol ip u32 match ip protocol 17 0xff - (also) match UDP packets
- protocol ip u32 match ip protocol 0x6 0xff - match TCP packets
- protocol ip u32 match ip protocol 1 0xff - match ICMP (ping) packets
- protocol ip u32 match ip dst 4.3.2.1/32 - match all IP traffic headed for IP 4.3.2.1
- protocol ip u32 match ip src 4.3.2.1/32 match ip sport 80 0xffff - match all IP traffic from 4.3.2.1 port 80
- protocol ip u32 match ip sport 53 0xffff - match originating DNS (both TCP and UDP)
- protocol ip u32 match ip dport 53 0xffff - match response DNS (both TCP and UDP)
- protocol ip u32 match ip protocol 6 0xff match u8 0x10 0xff at nexthdr+13 - match packets with ACK bit set
- protocol ip u32 match ip protocol 6 0xff match u8 0x10 0xff at nexthdr+13 match u16 0x0000 0xffc0 at 2 - packets less than 64 bytes in size with ACK bit set
- protocol ip u32 match ip tos 0x10 0xff - match IP packets with "type of service" set to "Minimize delay"/"Interactive"
- protocol ip u32 match ip tos 0x08 0xff - match IP packets with "type of service" set to "Maximize throughput"/"Bulk" (see "QDISC PARAMETERS" in tc-prio man page)
- protocol ip u32 match tcp dport 53 0xffff match ip protocol 0x6 0xff - match TCP packets heading for dest. port 53 (my not work)
