The COVID-19 pandemic has led people to wear face masks daily in public. Although the effectiveness of face masks against viral transmission has been extensively studied, there have been few reports on potential hygiene issues due to bacteria and fungi attached to the face masks. We aimed to (1) quantify and identify the bacteria and fungi attaching to the masks, and (2) investigate whether the mask-attached microbes could be associated with the types and usage of the masks and individual lifestyles. We surveyed 109 volunteers on their mask usage and lifestyles, and cultured bacteria and fungi from either the face-side or outer-side of their masks. The bacterial colony numbers were greater on the face-side than the outer-side; the fungal colony numbers were fewer on the face-side than the outer-side. A longer mask usage significantly increased the fungal colony numbers but not the bacterial colony numbers. Although most identified microbes were non-pathogenic in humans; Staphylococcus epidermidis, Staphylococcus aureus, and Cladosporium, we found several pathogenic microbes; Bacillus cereus, Staphylococcus saprophyticus, Aspergillus, and Microsporum. We also found no associations of mask-attached microbes with the transportation methods or gargling. We propose that immunocompromised people should avoid repeated use of masks to prevent microbial infection.