We carry out timing and spectral studies of the Be/X-ray binary pulsar GX 304-1 using NuStar and XMM-Newton observations. We construct the long-term spin period evolution of the pulsar which changes from a long-term spin-up ($\sim1.3 \times 10^{-13}$ Hz s$^{-1}$) to a long-term spin-down ($\sim-3.4 \times 10^{-14}$ Hz s$^{-1}$) trend during a low luminosity state ($\sim10^{34-35}$ erg s$^{-1}$). A prolonged low luminosity regime ($L_X \sim 10^{34-35}$ erg s$^{-1}$) was detected during 2005–2010 and spanning nearly five years since 2018 December. The XMM-Newton and NuStar spectra can be described with a power law plus blackbody model having an estimated luminosity of $\sim2.5 \times 10^{33}$ and $\sim3.6 \times 10^{33}$ erg s$^{-1}$, respectively. The inferred radius of the blackbody emission is about 100–110 m which suggests a polar-cap origin of this component. From long-term ultraviolet observations of the companion star, an increase in the ultraviolet signatures is detected preceding the X-ray outbursts. The spectral energy distribution of the companion star is constructed which provides a clue of possible UV excess when X-ray outbursts were detected from the neutron star compared to the quiescent phase. We explore plausible mechanisms to explain the long-term spin-down and extended low luminosity manifestation in this pulsar. We find that sustained accretion from a cold disc may explain the prolonged low luminosity state of the pulsar since December 2018 but the pulsar was undergoing normal accretion during the low luminosity period spanning 2005–2010.