And that is precisely why paul said if there was no one present to interpret (by either the gift of interpretation of various tongues, or by someone present who actually spoke that tongue), then you were to remain silent in the gathering. Because you certainly aren't building anyone up if no one can understand a word you're saying.
So it is similar to him saying to eat at home first so that others won't go hungry. Edify yourself at home first, speaking in tongues, but don't insist on edifying and gratifying and feeding yourself in the gathering, making others go hungry!