There is a filter to return groups for each user. As we are using that filter to return all groups so that they can send message to any group. That is why we cannot restrict them to add themselves from a certain (in this case admin) group.
We can achieve any one of these 2 conditions.
Hope you understand.