The domain of smart home environments is viewed as a key element of the future Internet, and many homes are becoming “smarter” by using Internet of Things (IoT) technology to improve home security, energy efficiency and comfort. At the same time, enforcing privacy in IoT environments has been identified as one of the main barriers for realizing the vision of the smart home. Based on the results of a risk analysis of a smart home automation system developed in collaboration with leading industrial actors, we outline the first steps towards a general model of privacy and security for smart homes. As such, it is envisioned as support for enforcing system security and enhancing user privacy, and it can thus help to further realize the potential in smart home environments.