Understanding PCIe performance for end host networking


Type
Conference Object
Change log
Authors
Neugebauer, Rolf 
Zazo, Jose Fernand 
Audzevich, Yury 
Lopez-Buedo, Sergio 
Abstract

In recent years, spurred on by the development and availability of programmable NICs, end hosts have increasingly become the enforcement point for core network functions such as load balancing, congestion control, and application specific network offloads. However, implementing custom designs on programmable NICs is not easy: many potential bottlenecks can impact performance.

This paper focuses on the performance implication of PCIe, the de-facto I/O interconnect in contemporary servers, when interacting with the host architecture and device drivers. We present a theoretical model for PCIe and pcie-bench, an open-source suite, that allows developers to gain an accurate and deep understanding of the PCIe substrate. Using pcie-bench, we characterize the PCIe subsystem in modern servers. We highlight surprising differences in PCIe implementations, evaluate the undesirable impact of PCIe features such as IOMMUs, and show the practical limits for common network cards operating at 40Gb/s and beyond. Furthermore, through pcie-bench we gained insights which guided software and future hardware architectures for both commercial and research oriented network cards and DMA engines.

Description
Keywords
PCIe, reconfigurable hardware, Operating System
Journal Title
SIGCOMM '18 Proceedings of the 2018 Conference of the ACM Special Interest Group on Data Communication
Conference Name
ACM SIGCOMM 2018
Journal ISSN
Volume Title
Publisher
Association for Computing Machinery
Sponsorship
Engineering and Physical Sciences Research Council (EP/P025374/1)