Understanding PCIe performance for end host networking

Conference Object
Change log
Neugebauer, Rolf 
Zazo, Jose Fernand 
Audzevich, Yury 
Lopez-Buedo, Sergio 

In recent years, spurred on by the development and availability of programmable NICs, end hosts have increasingly become the enforcement point for core network functions such as load balancing, congestion control, and application specific network offloads. However, implementing custom designs on programmable NICs is not easy: many potential bottlenecks can impact performance.

This paper focuses on the performance implication of PCIe, the de-facto I/O interconnect in contemporary servers, when interacting with the host architecture and device drivers. We present a theoretical model for PCIe and pcie-bench, an open-source suite, that allows developers to gain an accurate and deep understanding of the PCIe substrate. Using pcie-bench, we characterize the PCIe subsystem in modern servers. We highlight surprising differences in PCIe implementations, evaluate the undesirable impact of PCIe features such as IOMMUs, and show the practical limits for common network cards operating at 40Gb/s and beyond. Furthermore, through pcie-bench we gained insights which guided software and future hardware architectures for both commercial and research oriented network cards and DMA engines.

PCIe, reconfigurable hardware, Operating System
Journal Title
SIGCOMM '18 Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication
Conference Name
Journal ISSN
Volume Title
Association for Computing Machinery
Engineering and Physical Sciences Research Council (EP/P025374/1)