There is plenty of talk about Bonding and Multipath IO, but it is very difficult to get solid information about either one. Typically what documentation can be found is very bulky and the most important practical questions go unanswered.
As a result the following questions are often heard:
- When should I use bonding and when should I use multipath?
- I was expecting better throughput with bonding, why am I not seeing this?
- My RAID array shows 400MB/sec with local test, how can I get 400/MB sec outside?
Before we answer the above questions, let’s understand first how MPIO and bonding works.
MPIO allows a server with multiple NICs to transmit and receive I/O across all available interfaces to a corresponding MPIO-enabled server. If a server has two 1Gb NICs and the storage server has two 1Gb NICs, the theoretical maximum throughput would be about 200 MB/s.
Link aggregation (LACP, 802.3ad, etc.) via NIC teaming does not work the same way as MPIO. Link aggregation does not improve the throughput of a single I/O flow. A single flow will always traverse only one path. The benefit of link aggregation is seen when several “unique” flows exist, each from different source. Each individual flow will be sent down its own available NIC interface which is determined by a hash algorithm. Thus with more unique flows, more NICs will provide greater aggregate throughput. Link aggregation will not provide improved throughput for iSCSI, although it does provide a degree of redundancy.
Bonding works between a server and switch. Numerous workstations using each using a single NIC connected to the switch will benefit from bonded connections between the switch and storage server.
MPIO works between a storage server and the client server, whether or not there is a switch in the path.
With these basics fact, it will now be easier to answer our questions.
Q: When do I need bonding, and when is multipath appropriate?
A: Bonding works for a NAS server with multiple workstations connected.
MPIO works between hosts and initiators on FC or iSCSI. An example of MPIO configuration with a performance test showing 200MB/sec using dual Gb NIC’s is demonstrated step-by-step at: How to configure DSS V6 MPIO with Windows 2008 Server .
In short:
- Bonding works for NAS
- MPIO works for SAN
Q: I was expecting better throughput with bonding, in my case why does it not work?
A: Let us consider 4 workstations and a storage server. Bonding will increase performance only if the single 1 GB connection between the storage server and switch is a bottleneck. This can happen when total aggregate benchmark of all four workstations is a bit above 100 MB/sec. In such a case it is possible that the RAID array can move more data and the bottleneck is the single 1Gb NIC. If we add a second NIC and create bonding the total aggregate performance may improve. This will happen if the RAID array is fast enough to move more than 100MB/sec with four streams. However, as more streams are active, the RAID array performance may drop because the performance test pattern will be closer to random and not sequential. The sequential test pattern is valid for one single data stream only. In a sequential pattern performance test we can observe a very high data rate, but in random patterns it can be very low. This is because the hard disks need to make frequent seeks. In practical applications it is very difficult to make effective use of more than 2 NICs for bonding.
Q: My RAID array shows 400MB/sec with a local sequential benchmark test, how can I get 400/MB sec outside?
A: It is rather impossible to get these kinds of results with NAS and bonding, because bonding works with multiple workstations and this will result in predominantly random patterns. The same local test but with random patterns may drop well below 100MB/s and then even a single NIC will be not the bottleneck – instead it will be the RAID array (hard disks). With MPIO we can keep sequential pattern performance because it can operate with a single data flow. Thus, with 2 NICs in MPIO we can observe 200 MB/sec, with 4 NIC’s, 400MB/sec.
10 Comments
Falk Duebbert /
12, 12 2012 11:24:11We have a pair of two DSS V6 with 6 NICs each. Until now we’re running 4 NICs MPIO, one for replication and one for administration. Would bonding 2 NICs for replication increase our write-speed?
Kasia Kolodziej /
15, 01 2013 09:21:03Great question.
Yes, bonding 2 NICs for replication could increase your writing speed, but:
a) if they are directly connected : use balance-rr
b) if they are all connected to the same switch, use balance-rr and configure port trunking on the switch
c) if the switch supports LACP, you may use 802.3ad and configure ports on the switch
d) if ports are connected to different switches, you may use balance-alb
If you have any further questions, please contact our support team at pre-sales@open-e.com and they will gladly help you.
Gary Kirk /
13, 05 2013 05:25:44What’s up to all, I am in fact keen of reading this webpage’s post. It carries nice stuff.
Alan /
25, 07 2015 08:45:09I am not sure if this was the case when the article was written, but with regards to using bonding-rr, if you put each NIC on its own VLAN, you can achieve the increased speeds.
Damian /
05, 06 2016 07:34:06MPIO vs. LACP? I see alot of talk about the difference between the two, but I don’t see much about coexistence. Can MPIO and LACP be run on the same set of nics ports simultaneously? From what I understand LACP can be run on layer 2, and mpio layer 3. Can’t the two just get along and run on the same set? iSCSI devices using ips assigned to individual cards and lacp on one grouped lan ip?
karolina.pletnia /
15, 06 2016 01:08:26Hello Damian!
We do not expose the NIC IP in the bond, only the bond IP itself. MPIO creates multiple I/O streams where a bond is only one and both provide path failover. MPIO is for SANs (iSCSI and FC) and Bonds are used for NAS (File based).