[tex](D^2-1)y=x^2\sin x[/tex]
First consider the homogeneous part,
[tex](D^2-1)y=\dfrac{\mathrm d^2y}{\mathrm dx^2}-y=0[/tex]
which has characteristic equation
[tex]r^2-1=(r-1)(r+1)=0[/tex]
This has roots [tex]r=\pm1[/tex], so the characteristic solution is
[tex]y_c=C_1e^x+C_2e^{-x}[/tex]
For the nonhomogeneous part, consider a solution of the form
[tex]y_p=(a_2x^2+a_1x+a_0)\sin x+(b_2x^2+b_1x+b_0)\cos x[/tex]
which has second derivative
[tex]\dfrac{\mathrm d^2y_p}{\mathrm dx^2}=(-b_2x^2+(4a_2x-b_1)x+2a_1-b_0+2b_2)\cos x+(a_2x^2+(a_1+4b_2)x+a_0-2a_2+2b_1)\sin x[/tex]
Substituting into the ODE gives
[tex](-2b_2x^2+(4a_2-2b_1)x+2a_1-2b_0+2b_2)\cos x+(-2a_2x^2+(-2a_2-4b_2)x-2a_0-2b_1+2a_2)\sin x=x^2\sin x[/tex]
Matching up coefficients gives the system
[tex]\begin{cases}-2b_2=0\\4a_2-2b_1=0\\2a_1-2b_0+2b_2=0\\-2a_2=1\\-2a_1-4b_2=0\\-2a_0-2b_1+2a_2=0\end{cases}[/tex]
which has solutions
[tex]a_2=-\dfrac12,a_1=0,a_0=\dfrac12,b_2=0,b_1=-1,b_0=0[/tex]
So the particular solution is
[tex]y_p=\left(-\dfrac12x^2+\dfrac12\right)\sin x-x\cos x[/tex]
Therefore the general solution is
[tex]y=y_c+y_p[/tex]
[tex]y=C_1e^x+C_2e^{-x}+\left(-\dfrac12x^2+\dfrac12\right)\sin x-x\cos x[/tex]
