The aim is to investigate the function and outcomes of a residential narrative of “us and “them” connected to the categories “Swedes” and “immigrants”. Norbert Elias’s figurational sociology is used for analysing the establishment and maintenance of power relations through the narrative, in a city in south of Sweden. The results indicate that by controlling the narrative, the more established residents can strengthen their position in a residential figuration by blaming newcomers for any and all problems. This is possible through a strong social cohesion, based on shared common memories of the past, including denial of previous antagonism, and the concepts of “almost Swedes” and “having done your bit”. The results also indicate that residents in different categories are active in the reproduction of the narrative and its power relations. Crucial to the maintenance is the distancing process engendered between newcomers.