Theres an option in your settings for google calendar that is something along the lines of "Automatically add invitations to Calendar" that you can turn off, and I think it will fix that problem for you.