Yeah, pretty much. So in the case of a 1:4 mixture you essentially have 40% racemic material and 60% of one enantiomer in excess.
The equation you need is alpha = [alpha]*c*l
alpha is the observed rotation, [alpha] is the specific rotation, c is the concentration of the sample, and l is the pathlength of the instrument.
[alpha] won't change because you have the same compound and all that. You assume that l is the same too. So what has changed? The effective c is different because only 60% of the material will be productively rotating the plane polarized light (the racemic material doesn't do anything). So if the effective c is decreased from 1 (arbitrarily) to 0.6, then the observed rotation should decrease from 100 to 60. So in essence the value of c is artificially high because you have a equal amount of material (10 mg in each sample) but in one sample all of the material is rotating light whereas in the other only 60% is rotating light.
I think that's right, but I'm not really that experienced in dealing with optical rotations.