Assessing credibility of a q-q plot by presence of outliers

What proportion of a data set should significantly betray a particular
distribution in order to reject the hypothesis that this is the correct
distribution (i.e. how many points in a data set of n=100, say, should
lie significantly away from the diagonal line in a q-q plot)?

I have a q-q plot that seems to have all the hallmarks of a correct
fit, but there are data points in upper quantiles that don't fit the
line. Are there any web references on this topic?

Thanks for any help you can give.