Structure of the linearized gravitational Vlasov-Poisson system close to a polytropic ground state