The hardest part is that you sold two games to get this one so if you let him have the game back, you're now out two games. Granted, you have the money from the sale of the two games and you'll get the money back from the game you bought but if you don't have another game you'd want then you're really out 3 games (the 2 you sold and the one you're now being asked to give back). I think I'd tell him that you're sorry but you sold games to get this one and that you're going to keep it. If you ever get to the point where you want to sell it, you can check with him first to see if he wants to buy it back.
Seller's remorse is a tough lesson to learn but you're in the right to keep the game if you'd like.