Prior research shows that women’s higher fear of crime compared with men can largely be explained by women’s fear of rape. Whether women’s higher fear can also be explained by fear of nonviolent sexual harassment has not been explored. This study tested the shadow of sexual assault hypothesis in a sample of almost 3,000 Swedish university students. Our results confirm previous tests of the shadow thesis on the effect of fear of rape. In addition, we show that fear of sexual harassment also explained differences in fear between men and women. Based on the findings, we recommend that strategies to reduce sexual violence should focus on the entire continuum of violence against women.