We investigate the effect of turbulent magnetic fields on the observed spectral properties of synchrotron radio emission in large-scale radio galaxy lobes. We use three-dimensional relativistic magnetohydrodynamic simulations of fast, high-powered jets to study the structure of the lobe magnetic fields and how this structure affects the radio spectrum of the lobes. It has previously been argued that lobe ages inferred from radio spectra underestimate the true ages of radio galaxies due to re-acceleration of electrons in the lobe, mixing of electron populations, or the presence of turbulent magnetic fields in the lobes. We find that the spectral ages with and without accounting for the lobe magnetic field structure are consistent with each other, suggesting that mixing of radiating populations of different ages is the primary cause of the underestimation of radio lobe ages. By accounting for the structure of lobe magnetic fields, we find greater spectral steepening in the equatorial regions of the lobe. We demonstrate that the assumptions of the continuous injection, Jaffe-Perola, and Tribble models for radio lobe spectra do not hold in our simulations, and we show that young particles with high magnetic field strengths are the dominant contributors to the overall radio lobe spectrum.