A method for semantic segmentation of microstructure evolution from 4D imaging data is described and demonstrated. The method is based on a joint histogram describing the time history of the grayscale in each voxel of the images. After identifying and labeling clusters in the joint histogram, the labels are mapped back to the image. The results demonstrate accurate segmentation and characterization of sample evolution. The advantages of the proposed method include automatic segmentation of many time steps and the ability to track grayscale evolution over time and thereby discriminate similar evolution in different material phases. The method is demonstrated through application to 4D X-ray tomography datasets of temperature cycling in cement mortar and tensile testing of a cast iron sample. Water and air exchange in a pore inside the cement mortar is successfully segmented as a function of temperature. In the case of the deforming cast iron sample, several damage mechanisms are identified and segmented. The method is implemented in an open-source Python package called evoSegment.