The main objective of this paper was to develop a mesostructure-based finite element model of rubber modified asphalt mixture to predict both the dynamic modulus master curve and phase angle master curve under a large frequency range. The asphalt mixture is considered as a three-phase material consisting of aggregate, asphalt mortar, and air void. The mesostructure of the asphalt mixture was digitized by a computed tomography (CT) scan and implemented into finite element software. The 2S2P1D model was used to obtain the viscoelastic information of an asphalt mortar under a large range of frequencies and temperatures. The continuous spectrum of the 2S2P1D model was converted to a discrete spectrum and characterized by the generalized Maxwell model for numerical simulation. The Prony series parameters of the generalized Maxwell model and the elastic modulus of the aggregates were inputted into the finite element analysis as material properties. The dynamic modulus tests of a rubber modified asphalt mortar and asphalt mixture were conducted under different temperatures and loading frequencies. The dynamic modulus master curve and phase angle master curve of both asphalt mortar and asphalt mixture were constructed. The frequency of the finite element simulations of the dynamic modulus tests ranged from 10−6 to 104. The dynamic modulus and phase angle of the asphalt mixture was calculated and the master curves were compared with the master curves obtained from the experimental data. Furthermore, the effect of the elastic modulus of aggregates on the master curves was analyzed. Acceptable agreement between dynamic modulus master curves obtained from experimental data and simulation results was achieved. However, large errors between phase angle master curves appeared at low frequencies. A method was proposed to improve the prediction of the phase angle master curve by adjusting the equilibrium modulus of the asphalt mortar.