I unfortunately have the same feeling. Around Iphone 4 or so (when they were a lot smaller and sleeker) I got a slim phone case that could hold a card and my ID. It felt so nice going on a trip with only that in my pocket (not having to worry about if I feel my wallet in my back pocket, etc) but I couldn’t get over the fact that if I lost the phone I would be utterly screwed and a long way from home. Tend to always have a back-up now adays since I don’t want to have to put up with that.

As well as this, GrapheneOS also supports automatic rebooting for if your phone is taken by force. As once you’ve logged in from a cold boot, your data is in a vulnerable state where cops can access it without needing your passcode. With GOS, you can specify an amount of time for the phone to wait since the most recent login, and once that time has passed, it will automatically reboot the phone, placing your data back into the cold and secure state, so that the cops must then acquire your passcode from you, at which point you’d be able to give them the duress pin and ensure that the data is removed safely.

.