Differences in coloration between the sexes (sexual dichromatism) can increase or decrease in a species through evolutionary changes in either or both sexes diverging or converging in their colors. Few previous studies, however, have examined the relative rates of such changes, particularly when dichromatism is lost. Using reflectance data from 37 species of the New World blackbird family (Icteridae), we compared evolutionary rates of plumage color change in males and females when dichromatism was either increasing (colors diverging) or decreasing (colors converging). Increases in dichromatism involved divergent changes in both sexes at approximately equal rates. Decreases in dichromatism, in contrast, involved changes in females to match male plumage colors that were significantly more rapid than any changes in males. Such dramatic changes in females show how selection can differ between the sexes. Moreover, these evolutionary patterns support the idea that losses of dimorphism involve genetic mechanisms that are already largely present in both sexes, whereas increases in dimorphism tend to involve the appearance of novel sex-specific traits, which evolve more slowly. Our results have broad implications for how sexual dimorphisms evolve.