Graphitization degree of carbon matrix in ZrC-modified carbon composites is crucial to mechanical and ablation properties of the materials. In order to investigate the effect of ZrC formation on graphitization of the carbon matrix, microstructure of the carbon phase was investigated by X-ray (XRD), Raman, and X-ray photoelectron spectroscopy (XPS) analysis of the ceramic products obtained from zirconium containing polymer precursors at different pyrolysis temperatures. Compared with pure carbon phase, significant increase of average crystal thickness and microcrystalline planar size was observed in the carbon phase of the ZrC–C ceramics, together with the decrease of interlayer spacing and integrated intensity ratio of D peak to the G peak, indicating a significantly increased graphitization degree during the formation of ZrC. With the increasing ZrC content, amorphous (A) carbon was reduced remarkably, while turbostratic (T) component and graphitic (G) carbon components were increased, showing a slight higher graphitization degree. Moreover, the formation of ZrC was the key “ice breaking” step to decrease the defects and improve structure order of the carbon matrix. And the graphitization was dramatically enhanced during carbothermal reduction to create ZrC by breaking the carbon structure. Furthermore, coarsening and aggregation of ZrC particles as a result of high-temperature heat-treatment at 2500 °C and a high content of ZrC exhibit some negative influences on the structure of the carbon phase in ZrC–C ceramics.