In this paper we elaborate on the process of designing usability evaluations of technology that is to be integrated into specific real work settings, and where technology is the main target for change. Our research question relates to both what to evaluate and how to evaluate, and the ambition is to support evaluation designers on a practical level. We propose an explorative, operational method for usability evaluation design, which makes the design process explicit and design decisions more tractable. The evaluation design method is illustrated by an empirical example of how to design a comparative usability evaluation study, in a health care setting. The method supports the designer on a conceptual level by describing the different levels in the evaluation design. On an operational level it supports the designers by defining transitions between levels. And, on the analytical level it provides a template for a usability matrix, which allows for an explorative approach of constructing the conceptual content as well as a structured way to define and refine usability measures. However, the method needs to be further tested in other settings.