Broadband temporal modes of pulsed optical fields have been recently recognized as very promising for photonic quantum information processing and time-frequency metrology. Exploiting their full potential demands efficient and flexible tools for their manipulation. Among the tools demonstrated surprisingly the most basic, a single-mode temporal filter, is missing. In this work, we propose an experimentally feasible approach to realize a genuine single-mode temporal filter that is based on the concept of temporal cavity, a device with temporal mode-dependent resonances in the basis of frequency comb modes. This functionality is achieved as temporal-domain analogy of spatial-mode cleaner cavities. This device will enable robust temporal mode filtering and detection, opening new prospects in time-frequency metrology and multidimensional quantum information processing.