With the continuous improvement of people’s living standards and their increasing demand for fresh food, the cold chain logistics industry has developed rapidly. One of the biggest challenges to today’s cold chain logistics is to offer fresh food while minimizing CO2 emissions. The fresh degree and CO2 emissions are involved in the vehicle routing optimization problem in the cold chain logistics. In order to meet the quality requirement for fresh agricultural products and low carbon logistics, a novel routing optimization model considering the costs of quality deterioration and carbon emissions (Low Carbon and Freshness Degrees Vehicle Routing Problem (LCFD-VRP)) for cold chain distribution was established in this study. This model takes into account the fixed cost, fuel cost and time window penalty cost. An improved ant colony algorithm (IACA) is used to optimize the whole vehicle distribution routing with its strong global search ability. Tabu Search (TS) algorithm is used to search the single vehicle distribution routing with its good local search ability. An IACA combined with TS (IACATS) was proposed to solve the above LCFD-VRP model. The practicability of the model and the effectiveness of the above improved algorithm are verified using a real case study. The results of Zhoushan Dayang Refrigerated Logistics Co., Ltd. showed that, compared with the traditional algorithm, IACATS could reduce the dispatching of two refrigerated vehicles, thus lowering the total cost by 4.94%, shortening the actual transportation distance by 5.50% and cutting the total CO2 emissions by 8.9%. Therefore, the LCFD-VRP model can effectively help to achieve the low carbon emissions, multi-variety and low-cost distribution of fresh agricultural products. The proposed model and IACATS algorithm would be used to optimize VRP in cold chain enterprises. The results of this study also provide management suggestions for cold chain enterprises to effectively balance economic cost and environmental cost.