If you exchange a couple of email messages with another GMail user, Google Talk assumes that the person at the other end is a known contact and will automatically add him (or her) to your friends list.
That's the reason why many of us have long list of buddies in Google Talk (watch this video).
If the list has grown too big, try Google Talk bankruptcy - remove all existing friends from Google Talk in one go and make a fresh start.
Google Talk Friends List Getting Out of Hand? Try Bankruptcy