The low-temperature crack of asphalt concrete is considered to be one of the main deteriorations in asphalt pavements. However, there have been few studies on the composite effects of the aggregate characteristics and fracturing modes on the low-temperature cracking of asphalt concrete. Hence, the edge cracked semi-circular bend tests and the discrete element modeling approaches are combined to investigate the effect of the aggregate contents, aggregate morphological features and aggregate distributions on the fracture behavior of asphalt concrete in different fracturing modes at different temperatures. The results show that the fracture toughness and the crack extended time reduce with the increasing aggregate orientation and flatness and the decreased aggregate content. The effect of aggregate flatness is nonlinear and its reduction trend grows gradually with the increasing flatness. The total number of failed contacts is reduced with the increasing aggregate orientation and flatness, particularly at 10 °C. The number of failed contacts that occurred in the aggregate-mastic interface in quasi mode II fracturing is slightly higher than that in other fracturing modes. The aggregate distribution in the crack initiation zone greatly influenced the crack resistance, particularly at 10 °C. The research is beneficial to better understanding the fracture mechanisms of asphalt concrete at low-temperature.