This article examines how administrators in social services (n = 70) and habilitation staff in healthcare (n = 40) in Sweden experience difficulties in their work with people with intellectual disabilities. The research aim was to investigate the most typical aspects of difficult working situations and to apply a comparative analysis of differences and similarities, where the respondents' organizational affiliations are taken into account. The results are primarily based on a content analysis. Contextual standardized questions were included in this study. The results revealed that experiences of difficulties were categorized in four typological themes: difficult situations associated with (1) structure, (2) professional role, (3) relationships, and (4) collaboration. The respondents' experiences of difficult situations in social services and healthcare organizations did not correspond to the respondents' work dissatisfaction or unclear goals. On the contrary, handling this plurality within a specific organizational context was a fundamental aspect of professionalism. The different characteristics of the organizations in this study reflected two different institutional logics. While the administrators mainly operated within an administrative logic based on a regulatory framework, the habilitation staff operated within a therapeutic logic based on a cognitive framework. Consequently, the two groups had their own specific norms and rule systems, which influenced when, and to what extent, everyday situations were experienced and defined as difficult. The organizational context seems to influence experiences of difficult situations in the work with people with ID and this calls for a discussion of how it impacts the quality of services within intellectual disability services.