An abundant amount of data gathered during wind tunnel testing and health monitoring of structures inspires the use of machine learning methods to replicate the wind forces. This paper presents a data-driven Gaussian Process-Nonlinear Finite Impulse Response (GP-NFIR) model of the nonlinear self-excited forces acting on structures. Constructed in a nondimensional form, the model takes the effective wind angle of attack as lagged exogenous input and outputs a probability distribution of the forces. The nonlinear input/output function is modeled by a GP regression. Consequently, the model is nonparametric, thereby circumventing to set up the function's structure a priori. The training input is designed as random harmonic motion consisting of vertical and rotational displacements. Once trained, the model can predict the aerodynamic forces for both prescribed input motion and aeroelastic analysis. The concept is first verified for a flat plate's analytical solution by predicting the self-excited forces and flutter velocity. Finally, the framework is applied to a streamlined and bluff bridge deck based on Computational Fluid Dynamics (CFD) data. The model's ability to predict nonlinear aerodynamic forces, flutter velocity, and post-flutter behavior are highlighted. Applications of the framework are foreseen in the structural analysis during the design and monitoring of slender line-like structures.