IP_DUMMYNET Back to my home pagedummynet 1. Descriptiondummynet is a flexible tool originally designed for testingnetworking protocols, and since then (mis)used for bandwidthmanagement.It simulates/enforces queue and bandwidth limitations, delays,packet losses, and multipath effects. It also implementsa variant of Weighted Fair Queueing called WF2Q+. It can be used on user'sworkstations, or on FreeBSD machines acting as routers or bridges.Just to get the idea of what you can do with dummynet, e.g. byusing dummynet on your workstation, or putting a PC with two ethernetcards between your network and your router and booting from the floppy-image below, here are a few examples :These rules limit the total ICMP traffic (inbound+outbound)to 50Kbit/s ipfw add pipe 1 icmp from any to any ipfw pipe 1 config bw 50Kbit/s queue 10These rules limit inbound traffic to 300Kbit/sfor each host on your network 10.1.2.0/24. ipfw add pipe 2 ip from any to 10.1.2.0/24 ipfw pipe 2 config bw 300Kbit/s queue 20 mask dst-ip 0x000000ffIf you want all machines to share evenly a single link, you shoulduse instead: ipfw add queue 1 ip from any to 10.1.2.0/24 ipfw queue 1 config weight 5 pipe 2 mask dst-ip 0x000000ff ipfw pipe 2 config bw 300Kbit/sAnd these rules simulate an ADSL link to the moon: ipfw add pipe 3 ip from any to any out ipfw add pipe 4 ip from any to any in ipfw pipe 3 config bw 128Kbit/s queue 10 delay 1000ms ipfw pipe 4 config bw 640Kbit/s queue 30 delay 1000msdummynet works by intercepting packets (selected by ipfwrules - ipfw is one of the FreeBSD firewalls) in their way throughthe protocol stack, and passing them through one or more objectscalled queues and pipes, which simulate the effects of bandwidthlimitations, propagation delays, bounded-size queues, packet losses,multipath. Pipes are fixed-bandwidth channels. Queues representinstead queues of packets, associated with a weight, which sharethe bandwidth of the pipe they are connected to proportionallyto their weight.Each pipe and queue can be configured separately, so you can apply differentlimitations/delays to different traffic according to the ipfw rules(e.g. selecting on protocols, addresses and ports ranges, interfaces,etc.).Pipes and queues can be created dynamically, so using a single set of rulesyou can apply independent limitations to all hosts in a subnet, orto all types of traffic, etc.You can also configure the system to build cascades of pipes, soyou can simulate networks with multiple links and paths betweensource(s) and destination(s).2. Performance, status and availabilityUnlike other traffic shaping packages which run in userland,dummynet has a very little overhead, asall processing is done within the kernel. There isno data copying involved to move packets through pipes,just a bit of pointer shuffling, and the implementation isable to handle thousands of pipes with O(log N) cost, where Nis the number of active pipes.The WFQ variant we implement, called WF2Q+, has a complexitywhich is O(log N) in the number of active flows, so again itis able to handle efficiently thousands of flows.dummynet is part of FreeBSDsince Sept.1998. It has been recently (Jan.2000 and June 2000) rewritten, so themost recent, feature-rich and robust versions are in FreeBSD3.4-STABLE and newer releases.You don't need to install FreeBSD on your hard disk to use it, asbelow you will find a bootable single-floppyversion of FreeBSD which includes dummynet, bridging, and a lot ofother goodies.Dummynet is being heavily used by lots of people, and the codeseems to be extremely stable and robust, especially in the3.4-STABLE version and above. Bug fixes are generally applied to theFreeBSD source tree and are available from theCVS treeor in newer snapshot/releases of FreeBSD.From time to time i update the floppy image on this site as well.3. SupportIf you have found some bug, please report it to meby email, but don't forget to include information on whichversion of FreeBSD and dummynet you are using, your rules(ipfw show; ipfw pipe show), your configuration(bridge or router) etc.If you have a simple question, again just email me and i generally tryto reply as soon as possible. Again, please supply details!For more complex things (like "i haveno time to learn how to use it, i just want this work done"), orcustomizations and additions of new features to dummynet/ipfw, Iam available (through my department) for doing support on a contractbasis.Email luigi@iet.unipi.itfor discussing details.This said, FreeBSD users should be able to use dummynet withoutthe need for support.The relevant manpages (ipfw(8), dummynet(4), bridge(4))are a great source of information, so please read updatedversion of them before asking questions.You can also try postingon the various FreeBSD mailing lists or newsgroups, they areusually a very good source of information.4. Using dummynetDummynet is entirely controlled by the ipfw commandsand a set of sysctl variables.4.1 Basic ipfw commandsThe basic structure of ipfw commands is ipfw add [N] [prob X] action PROTO from SRC to DST [options]where N is the rule number ;X is a number between 0 and 1 that, when present, indicatesthe probability of getting a match on this ruleif all other fields are correct. The default is deterministicmatch;action is one of the actions executed on a match,which can be any of allow, deny, skipto N, pipe Nand others. To send a packet to a dummynet pipe, we haveto use pipe N;PROTO is the protocol type we want to match (IP, TCP, UDP, ...);SRC and DST are address specifier (we can use addresses with netmasksand optionally followed by ports or port ranges);options can be used to restrict the attention topackets coming from/to specific interfaces, or carrying someTCP flags or ICMP options, or bridged, etc.4.2 Sysctl variablesThe following are the main sysctl variables to control the behaviour of ipfw, bridging and dummynet:Controlling ipfwThe firewall is mostly controlled by ipfw, and thesysctl variables only serve to give global configurationand default parameters.net.inet.ip.fw.enable: 1 enables firewall in the IP stacknet.inet.ip.fw.one_pass: 1 Forces a single pass through the firewall. If set to 0, packets coming out of a pipe will be reinjected into the firewall starting with the rule after the matching one. NOTE: there is always one pass for bridged packets.net.inet.ip.fw.dyn_buckets: 256 (readonly) Current hash table size used for dynamic rules.net.inet.ip.fw.curr_dyn_buckets: 256 Desired hash table size used for dynamic rules.net.inet.ip.fw.dyn_count: 3 Current number of dynamic rules. (readonly) net.inet.ip.fw.dyn_max: 1000 Max number of dynamic rules. If you exceed this limit, you will have to wait for a rule to expire before being able to create a new one.net.inet.ip.fw.dyn_ack_lifetime: 300net.inet.ip.fw.dyn_syn_lifetime: 20net.inet.ip.fw.dyn_fin_lifetime: 20net.inet.ip.fw.dyn_rst_lifetime: 5net.inet.ip.fw.dyn_short_lifetime: 5 Lifetime (in seconds) for various types of dynamic rules.Controlling dummynetAlso dummynet is mostly controlled by ipfw, with thesysctl variables serving mostly for default parameters.net.inet.ip.dummynet.hash_size: 64 Size of hash table for dynamic pipes.net.inet.ip.dummynet.expire: 1 Delete dynamic pipes when they become empty.net.inet.ip.dummynet.max_chain_len: 16 Max ratio between number of dynamic queues and hash buckets. When you exceed (max_chain_len*buckets) queues on a pipe, packets not matching any of these will be all put into the same default queue.Controlling bridgingBridging is almost exclusively controlled by sysctl variables.net.link.ether.bridge_cfg: ed2:1,rl0:1, set of interfaces for which bridging is enabled, and cluster they belong to.net.link.ether.bridge: 0 enable bridging.net.link.ether.bridge_ipfw: 0 enable ipfw for bridging.4.3 Pipe and queue configurationThe following ipfw commands control dummynet pipesipfw pipe NN config ...This command is used to create or reconfigure a pipe.NN is the numericidentifier (between 1 and 65535) of the pipe. Issuing multipletime the configuration command results in the pipe beingreconfigured.ipfw [-s field] pipe [NN] showThis command shows the parameters of a pipe. If the pipe isa dynamic one (see mask parameter), then all dynamicpipes created from this one are listed. The list can be very verylong. The -s option allows you to sort the listing on oneof the four counters associated to the pipe.ipfw pipe NN deleteDestroys a single pipe. Remember that packets sent toa non-existing pipe are silently dropped.ipfw pipe flushDestroys all pipes.The following parameters can be configured for a pipe, adding thecommand in the pipe config... line:Bandwidth: bw NNunitNN is the bandwidth assigned to the pipe, unit (which mustfollow the number with no intervening spaces) can be any ofbit/s Kbit/s Mbit/s Byte/s KByte/s MByte/s ornon-ambiguous abbreviations.A bandwidth of 0 (or no bandwidth) results in no bandwidthlimitations (hence, no queues will ever build up).Queue size: queue NN [unit]Sets the queue size, in slots if only NN is specified,otherwise in Bytes or KBytes. When there is no room in thequeue, packets are dropped. The default queue size is 50 slots.The combination of bandwidthand queue size influence the queueing delay. Be carefulwhen using low bandwidths not to use too large queues, oryou might end up with several seconds of queueing delay.Also be careful when you specify the queue size in packets:if you run tests over the loopback interface, a packet canbe very large, e.g. 16KB, again resulting in huge delays.Delay: delay NN msSets the propagation delay of the pipe, in milliseconds.Note that the queueing delay component is independent ofthe propagation delay. Also note that all delays are approximatedwith a granularity of 1/HZ seconds (HZ is typically 100, but wesuggest using HZ=1000 and maybe even larger values).Random Packet Loss: plr XX is a floating point number between 0 and 1 which causespackets to be dropped at random. This is done generally tosimulate lossy links. The default is 0, or no loss.Dynamic queue creation: mask ...It is possible to associate a mask to a pipe so thatbandwidth and queue limitations are enforced separatelyfor packets belonging to different flows.The mask command lets you specify which partsof the following fields contribute to identify a flow:[proto N] [src-ip N] [dst-ip N] [src-port N] [dst-port N]where N is a bitmask where significant bits are set to 1.You can specify one or more masks, or the allkeyword to mean that all fields are fully significant.The default (when no mask are specified) is to ignoreall fields, so that all packets are considered to belongto the same flow.Whenever a new flow is encountered, a new queue (withthe specified bandwidth and queue size) is created.WARNING!!! the number of dynamic queues that canbe created in this way can become very large. They areaccessed through a hash table, whose size you can defineusing the buckets NN specifier after the mask command.To use WF2Q+, packets must be passed to queues which in turnmust be connected to a pipe.The following ipfw commands control dummynet pipesipfw queue NN config ...This command is used to create or reconfigure a queue.NN is the numericidentifier (between 1 and 65535) of the queue. Issuing multipletime the configuration command results in the queue beingreconfigured.ipfw queue NN deleteDestroys a single queue. Remember that packets sent toa non-existing queue are silently dropped.ipfw queue flushDestroys all queues.The following parameters can be configured for a queue, adding thecommand in the queue config... line:Pipe: pipe NNNN is the identifier of the pipe used for regulating traffic.Weight: weight NNNN is the weight (1..100, default 1)associated to the queue.Per-Flow queueing: mask ...The syntax is the same as for pipes. However, all queuescreated dynamically will share the parent pipe's bandwidth accordingto the weight.Queue size, Random Packet Loss:Same as for pipes.5. Using dummynet for testing protocolsDummynet was originally created to test network protocols andapplications, possibly even on a standalone system. As a consequence,some of its features such as delay emulation, random loss etc. areexplicitly designed for that purpose.There are a few things you should take in mind when doing suchtests, to avoid getting incorrect results. They are allobvious things, still it is better to have them in mind.Choosing a reasonable buffer size.As said earlier, packet can be subject to a delay which isproportional to the total queue size (in bytes), and inverselyproportional to the bandwidth. At low bandwidths, this queueingdelays can be extremely high, especially if the queue size isdefined in terms of packets and packets are large. The defaultqueue size is almost certainly too large for most purposes, andit is often preferable to define the queue size in terms of bytes rather than packets.Half-duplex vs. Full-duplex channels.With the exception of shared-medium networks such as theethernet, most links that you want to simulate for yourexperiments are full duplex. As such, the proper configurationis the following: ipfw add pipe 1 ip from A to B ipfw add pipe 2 ip from B to A ipfw pipe 1 config ... ipfw pipe 2 config ...Should you really need to mode a half duplex network,then you can use the following sequence. But think twice beforeyou do so, as it is often a non-realistic mode. ipfw add pipe 3 ip from A to B ipfw add pipe 3 ip from B to A ipfw pipe 3 config ...Interactions between bridging and multicastYou can use ipfw (and dummynet) in a bridge by setting somesysctl variables: sysctl -w net.link.ether.bridge=1 sysctl -w net.link.ether.bridge_ipfw=1and then specify your firewall configuration.Be careful when you run experiment involving multicast trafficthrough a dummynet-enabled bridge.Unless you set the rules right, multicast traffic ina bridge goes through the firewall code twice: onceduring forwarding at level 2, once when the packet ispassed to the local IP stack of the bridge.Starting from Feb.2000, there are to avoid this problem.One involves a sysctl variable: sysctl -w net.inet.ip.fw.enable=0which avoids that the firewall is invoked at the ip level.Otherwise, you can use the bridged specifier in yourruleset to match only bridged packets: ipfw add pipe 1 ip from any to any bridgedRunning over the loopback interface.Dummynet was originally designed for running experiments on astandalone machine. The loopback interface lets you runsenders and receivers on the same machine, but you shouldremember a few things:The firewall is invoked on all packets.This means that if you have a configuration such as ipfw add pipe 4 ip from 127.0.0.1 to 127.0.0.1 ipfw pipe 4 config delay 100msand do a simple ping 127.0.0.1 you will see a delayof approximately 400ms. In fact the ICMP request goes throughthe pipe twice (once down, once up), and the same for the ICMPreply. For the same reason, if you also have bandwidth or queuelimitations, remember that the queue sees the traffic multipletimes.You can partially overcome this problem by using additionalipfw options, e.g. specifying a directionfor matching packets, or the uid of the sender or receiving process.Alternatively, you can assign multiple aliases to the loopbackinterface, and make sure that the sender and receiver bindtheir local endpoint to different addresses so thatyou will have distinct rules matching traffic in thetwo directions.The MTU of the loopback interface defaults to 16KBThe usual default for ethernet is 1500, and for point-to-pointlinks often smaller (576 or so). You can simply fix this byredefining the mtu to the desired value with ifconfig lo0 mtu 1500TCP defaults.Be very careful when using TCP, especially between processes runningon the same machine, or on the same subnet.Apart from the MTU issue mentioned earlier, at least on FreeBSD,TCP starts with a full window when the remote endpoint is on thesame subnet as one of the local addresses.You need a simple fix in the source (tcp_input.c i believe)to fix this behaviour in FreeBSD 3.x, whereas FreeBSD 4.x hassysctl variable(s) to set the initial window.Secondly, when you do experiments on configuration witha large delay-bandwidth product, remember that many applicationsuse the default window size which is small, something like 16KB.You might end up not using the full bandwidth just because yourdata transfer is window-limited.5.1 Simulating multipathOne nice feature of the new version of dummynet is the ability tosimulate multiple paths between sender and receiver.This is done using probabilistic match, e.g.: ipfw add prob 0.33 pipe 1 ip from A to B ipfw add prob 0.5 pipe 2 ip from A to B ipfw add pipe 3 ip from A to B ipfw pipe 1 config ... ipfw pipe 2 config ... ipfw pipe 3 config ...Given the right packet, the first rule will match with probability 1/3;in the remaining 2/3 of occurrence we move to the second rule, whichwill match with prob 1/2 (so overall 1/2*1/3 = 1/3), and the remaining1/3 of occurrence will move to the third rule, which has adeterministic match. We can then configure the three pipes as desiredto emulate phenomena such as packet reordering etc.6 Related linksHere i collect some info on how to do various ipfw-relatedthings. Most of this is just URLs collected from the mailing listso the reliability of the info might be different (for good or bad)from what is in this page. ALTQAlternate queueing scheme.A PicoBSD floppyTo conclude... if you want to try dummynet, here is a bootablefloppy image of a system with FreeBSD, bridging, ipfw, dummynet,natd, ppp, drivers for a few interfaces, and accessible via telnet.To setup this system,download the 1.44MB image,pico.000608.bin and copy itto a floppy using dd under FreeBSD,or rawrite under DOS/Windows.Then put the floppy into a machine with hopefully at least oneinterface, and wait for it to boot. When the system comes up,login as root, password "setup", and you can play with bridging, ipfw and dummynet using the above commands.Luigi RizzoDipartimento di Ingegneria dell'Informazione -- Univ. di Pisavia Diotisalvi 2 -- 56126 PISAtel. +39-050-2217533 Fax +39-050-2217600email: rizzo@iet.unipi.it _uacct = "UA-2736043-1";urchinTracker(); |
|
