> If I knew that my contacts were being shared with a 3rd party (Signal), I would never have signed up.

I think, you should read up a bit on how Signal manages contacts/address books.

https://signal.org/blog/private-contact-discovery/