In this paper we consider linear Hamiltonian differential systems without thecontrollability (or normality) assumption. We prove the Rayleigh principle for thesesystems with Dirichlet boundary conditions, which provides a variational characterizationof the finite eigenvalues of the associated self-adjoint eigenvalue problem. This resultgeneralizes the traditional Rayleigh principle to possibly abnormal linear Hamiltoniansystems. The main tools are the extended Picone formula, which is proven here for thisgeneral setting, results on piecewise constant kernels for conjoined bases of theHamiltonian system, and the oscillation theorem relating the number of proper focal pointsof conjoined bases with the number of finite eigenvalues. As applications we obtain theexpansion theorem in the space of admissible functions without controllability and aresult on coercivity of the corresponding quadratic functional.
