Human milk oligosaccharides (HMOs) are multifunctional carbohydrates naturally present in human milk that act as prebiotics, prevent pathogen binding and infections, modulate the immune system and may support brain development in infants. HMOs composition is very individualized and differences in HMOs concentrations may affect the infant’s health. HMOs variability can be partially explained by the activity of Secretor (Se) and Lewis (Le) genes in the mother, but non-genetic maternal factors may also be involved. In this cross-sectional, observational study, 78 single human milk samples ranging from 17 to 76 days postpartum (median: 32 days, IQR: 25–46 days) were collected from breastfeeding Brazilian women, analyzed for 16 representative HMOs by liquid chromatography coupled to mass spectrometry and associations between maternal and infant factors with HMOs concentrations were investigated. HMOs concentrations presented a high variability even in women with the same SeLe phenotype and associations with maternal allergic disease, time postpartum and with infant’s weight, weight gain and sex. Overall, we present unprecedented data on HMOs concentrations from breastfeeding Brazilian women and novel associations of maternal allergic disease and infant’s sex with HMOs concentrations. Differences in HMOs composition attributed to maternal SeLe phenotype do not impact infant growth, but higher concentrations of specific HMOs may protect against excessive weight gain.