Although flavonoid phytoestrogens have been suggested to be associated with reduced risk of colorectal cancer (CRC), their influence on CRC prognosis remains uncertain. A population-based cohort of 2051 patients diagnosed with stage I–III CRC in southwest Germany in 2003–2010 were followed for five years. Post-diagnostic serum concentration of genistein and luteolin were measured using Ultra-Performance Liquid Chromatography with mass spectrometry. Multivariable Cox regression analysis was conducted to calculate the Hazard Ratios (HRs) and 95% confidence interval (CI) for the association between flavonoids concentration and overall morality, CRC-specific mortality, CRC recurrence, and disease-free survival (DFS). Median (interquartile range) serum concentration of genistein and luteolin was 11.90 ng/µL (10.08–14.13) and 7.20 ng/µL (6.40–8.16), respectively. Neither serum genistein nor luteolin was associated with CRC prognosis. There was no clear evidence of departure from linearity. However, the association might be differential by adjuvant chemotherapy. Associations pointed towards lower risk in patients who received chemotherapy and higher risk in those without chemotherapy for overall mortality regarding serum genistein (P-interaction = 0.02) and correspondingly for CRC recurrence (P-interaction: 0.03) and DFS (P-interaction: 0.01) with respect to luteolin. Our study provides little evidence that serum genistein and luteolin are associated with colorectal cancer prognosis. Future studies are warranted to evaluate the potential interaction with adjuvant chemotherapy.