Typical security advice is to turn off “find me by my number” in the Signal security settings. You may well have friends who use Signal, but have turned that setting off. You may consider turning it off yourself. Also, you should turn off “see my number” on the same settings screen.