You should have a +ve under the square root:
if Ka = x2/(0.1 - x)
then x2 + xKa - 0.1Ka = 0
and x = [-Ka + (Ka2 + 0.4Ka)1/2]/2
However, for a weak acid, such as anilinium, x << Concentration (0.1 in this case).
You can use the approximation: 0.1 - x ~ 0.1, which makes the calculation much simpler.