When there is noise in a covariance matrix, portfolio optimization tends to produce portfolios for which the risk forecasts are underestimates of the true risk. In this paper, we take a closer look at the connection between estimation error and the underestimation of the risk of optimized portfolios. We pay special attention to the case in which returns have a known factor structure. There, the bias in optimization can be reduced dramatically by using a covariance matrix based on a factor model, rather than one computed from historical asset covariances. Moreover, our analysis reveals that for many active portfolios, the bias in factor-model forecasts is less than previously thought. Lastly, we discuss the role of constraints in mitigating risk forecasting bias.