From the wide spectrum of potential applications of graphene, ranging from transistors and chemical sensors to nanoelectromechanical devices and composites, the field of photonics and optoelectronics is believed to be one of the most promising. Indeed, graphene's suitability for high-speed photodetection was demonstrated in an optical communication link operating at 10 Gbit s(-1). However, the low responsivity of graphene-based photodetectors compared with traditional III-V-based ones is a potential drawback. Here we show that, by combining graphene with plasmonic nanostructures, the efficiency of graphene-based photodetectors can be increased by up to 20 times, because of efficient field concentration in the area of a p-n junction. Additionally, wavelength and polarization selectivity can be achieved by employing nanostructures of different geometries.