A tame dynamical system can be characterized by the cardinality of its enveloping (or Ellis) semigroup. Indeed, this cardinality is that of the power set of the continuum $2^{\mathfrak c}$ if the system is non-tame. The semigroup admits a minimal bilateral ideal and this ideal is a union of isomorphic copies of a group $\mathcal H$, called the structure group. For almost automorphic systems, the cardinality of $\mathcal H$ is at most ${\mathfrak c}$ that of the continuum. We show a partial converse of this which holds for minimal systems for which the Ellis semigroup of their maximal equicontinuous factor acts freely, namely that the cardinality of $\mathcal H$ is $2^{{\mathfrak c}}$ if the proximal relation is not transitive and the subgroup generated by products $\xi \zeta ^{-1}$ of singular points $\xi ,\zeta $ in the maximal equicontinuous factor is not open. This refines the above statement about non-tame Ellis semigroups, as it locates a particular algebraic component of the latter which has such a large cardinality.