MSM - Can't Switch sites within certain Membergroups

Hello, I’m trying to make a Member Group that has certain permissions to access our Sub-Domain with MSM, but no access to any other domains.

The problem is, once I create that Member group, and login with a user in the group, and try to switch sites to our Sub Domain I get:

An Error Was Encountered You are not authorized to access this page

When logging in to the CP it’s automatically loaded our main site as the current site for the CP, when switching is when the error occurs.

Any ideas?