This thesis shows our investigation on scene understanding and object detection for surveillance applications, emphasizing context-aware computer vision models that enhance detection accuracy in complex environments while respecting privacy considerations. The research advances object detection by addressing key aspects such as variability across environments, contextual information, and multimodal data fusion. Through a comprehensive literature review, we examines the role of contextual information, such as spatial, scale, and temporal context, in improving detection performance. Furthermore, we introduce specialized object detection models designed for indoor and outdoor environments, demonstrating howscene-specific training enhances detection accuracy. We also explore hierarchical scene classification, analyzing how different levels contribute to scene recognition. Lastly, a multimodal fall detection method integrating video and audio is proposed, overcoming limitations of purely visual systems in obstructed or low-visibility conditions. The findings of all papers highlight the effectiveness of scene context, hierarchical classification, and multimodal fusion in developing robust, high-accuracy surveillance models suitable for real-world environments.