My parents always have cats to control rats/mouse or mice in the house otherwise they attack and spoil almost everything in the house.
There are also some mice (very tiny) that the cat would simply ignore. For those, my dad uses poison glue. He would stick them with some foods (foods that the mice are usually attracted to) and kept them in the house. That way, our house back at home was/is kept free from the menace of mice.
If you don't have the heart to kill but just want to catch them, then I guess you can still use non-poisonous glue to do it
I personally don't have any qualms killing them as I think that is the best way to keep them out forever. Otherwise, they will keep coming back either to your house or to somebody's house.