[tex]y=x^r[/tex]
[tex]\implies r(r-1)x^r+6rx^r+4x^r=0[/tex]
[tex]\implies r^2+5r+4=(r+1)(r+4)=0[/tex]
[tex]\implies r=-1,r=-4[/tex]
so the characteristic solution is
[tex]y_c=\dfrac{C_1}x+\dfrac{C_2}{x^4}[/tex]
As a guess for the particular solution, let's back up a bit. The reason the choice of [tex]y=x^r[/tex] works for the characteristic solution is that, in the background, we're employing the substitution [tex]t=\ln x[/tex], so that [tex]y(x)[/tex] is getting replaced with a new function [tex]z(t)[/tex]. Differentiating yields
[tex]\dfrac{\mathrm dy}{\mathrm dx}=\dfrac1x\dfrac{\mathrm dz}{\mathrm dt}[/tex]
[tex]\dfrac{\mathrm d^2y}{\mathrm dx^2}=\dfrac1{x^2}\left(\dfrac{\mathrm d^2z}{\mathrm dt^2}-\dfrac{\mathrm dz}{\mathrm dt}\right)[/tex]
Now the ODE in terms of [tex]t[/tex] is linear with constant coefficients, since the coefficients [tex]x^2[/tex] and [tex]x[/tex] will cancel, resulting in the ODE
[tex]\dfrac{\mathrm d^2z}{\mathrm dt^2}+5\dfrac{\mathrm dz}{\mathrm dt}+4z=e^{2t}\sin e^t[/tex]
Of coursesin, the characteristic equation will be [tex]r^2+6r+4=0[/tex], which leads to solutions [tex]C_1e^{-t}+C_2e^{-4t}=C_1x^{-1}+C_2x^{-4}[/tex], as before.
Now that we have two linearly independent solutions, we can easily find more via variation of parameters. If [tex]z_1,z_2[/tex] are the solutions to the characteristic equation of the ODE in terms of [tex]z[/tex], then we can find another of the form [tex]z_p=u_1z_1+u_2z_2[/tex] where
[tex]u_1=-\displaystyle\int\frac{z_2e^{2t}\sin e^t}{W(z_1,z_2)}\,\mathrm dt[/tex]
[tex]u_2=\displaystyle\int\frac{z_1e^{2t}\sin e^t}{W(z_1,z_2)}\,\mathrm dt[/tex]
where [tex]W(z_1,z_2)[/tex] is the Wronskian of the two characteristic solutions. We have
[tex]u_1=-\displaystyle\int\frac{e^{-2t}\sin e^t}{-3e^{-5t}}\,\mathrm dt[/tex]
[tex]u_1=\dfrac23(1-2e^{2t})\cos e^t+\dfrac23e^t\sin e^t[/tex]
[tex]u_2=\displaystyle\int\frac{e^t\sin e^t}{-3e^{-5t}}\,\mathrm dt[/tex]
[tex]u_2=\dfrac13(120-20e^{2t}+e^{4t})e^t\cos e^t-\dfrac13(120-60e^{2t}+5e^{4t})\sin e^t[/tex]
[tex]\implies z_p=u_1z_1+u_2z_2[/tex]
[tex]\implies z_p=(40e^{-4t}-6)e^{-t}\cos e^t-(1-20e^{-2t}+40e^{-4t})\sin e^t[/tex]
and recalling that [tex]t=\ln x\iff e^t=x[/tex], we have
[tex]\implies y_p=\left(\dfrac{40}{x^3}-\dfrac6x\right)\cos x-\left(1-\dfrac{20}{x^2}+\dfrac{40}{x^4}\right)\sin x[/tex]
