Here we consider the following fractional Hamiltonian system

\begin{equation*}\begin{cases}\begin{aligned}(-\Delta)^{s} u&=H_v(u,v) \;\;&&\text{in}~\Omega,\\(-\Delta)^{s} v&=H_u(u,v) &&\text{in}~\Omega,\\u &= v = 0 &&\text{in} ~ \mathbb{R}^N\setminus\Omega,\end{aligned}\end{cases}\end{equation*}

where $s\in (0,1)$, $N \gt 2s$, $H \in C^1(\mathbb{R}^2, \mathbb{R})$, and $\Omega \subset \mathbb{R}^N$ is a smooth bounded domain. To apply the variational method for this problem, the key question is to find a suitable functional setting. Instead of usual fractional Sobolev spaces, we use the solution space of $(-\Delta)^{s}u=f\in L^r(\Omega)$ for $r\ge 1$, for which we show the (compact) embedding properties. When H has subcritical and superlinear growth, we construct two frameworks, respectively with the interpolation space method and the dual method, to show the existence of nontrivial solution. As byproduct, we revisit the fractional Lane–Emden system, i.e. $H(u, v)=\frac{1}{p+1}|u|^{p+1}+\frac{1}{q+1}|v|^{q+1}$, and consider the existence, uniqueness of (radial) positive solutions under subcritical assumption.