Unable to Invite Other Sims to My Lot

When there are too many Family NPC that I'm not controlling, I can't invite other Sims to my lot anymore. When I try to invite them, they either don't come or leave the lot immediately after arriving. If I delete the selection_group.json file, the invitations work again, but the same issue reoccurs when the number of Family NPC increases again.