Plenty of studies show that the hormonal effects of phytoestrogen ingestion (e.g., lower levels of testosterone) could, in theory, result in infertility.
What we lack so far is evidence that this is a real problem in clinical practice. Even so, some level of suspicion may be warranted in infertile patients.