Well, you are right that the penalty applies to all attacks, but your analysis is wrong in that the CA bonus applies to all attacks that others make against you, which could be many attacks if, for example, , Surrounded. You can not determine its value by assuming that you will only be attacked once, I doubt someone will recommend the feat in that instance.
But all this said, the experience in combat is widely considered a useless feat. No, necessarily, due to the size of the bonus or the size of the penalty, but because AC is a defective defense. There are many other ways in which you can threaten a character who renounce his offense to improve only one factor of his defense is not a very wise idea.
In addition, the 3.5 tends to play at many levels, certainly in the first place, and then again at about the seventh, as a "rocket tag," where a successful offensive action usually eliminates a threat. This means that defenses must be absolute whenever possible. And if they are not, eliminating threats will normally be the safest option, the best defense is a good offensive.
This does not mean that there are no situations in which the combat experience is not pleasant. The real problem is that these situations are, or should certainly be, rare. And even wrestlers get very few exploits to spend on situational options. The characteristics can be extraordinarily powerful, and they can be built on each other in a synergistic way, and for the most part, it is extremely useless to do a feat that you will use in a weird way. You do not have enough deeds to do that.
Another way to see this is to compare feats against spells. Spellcasters get more spells massively than those who receive feats. A fighter of level 20 gets 18 dowries, while a magician prepares 18 spells from 7º. Spells are also much more powerful when you use them; There is no feat that ends a fight as a standard action, while there are many spells that will do it. The only advantage that exploits have is that they can be used all day. This is not a good enough advantage: spell castors above the lower levels should have little difficulty with their spells per day, as long as they are choosing good spells, but that's all warriors have. As such, it really is up to a warrior to make sure that his deeds are things they want to do all day. The combat experience is not that.
Add to this the fact that Combat Expertise can not be used in a fury, and a level or two of barbarians is extremely popular for almost all types of fighters thanks to the choice of the lion spirit totem in Full Champion (Provides pounce, the full attack capability after a charge, which makes it one of the few efficient ways to maintain offensive ability while moving). One rage per day means at least one fight in which you can not use the Combat Experience at all, and the rage is much better than the Combat Experience. That only lowers its value even more.
Therefore, the only reason most people take combat experience is because something, usually an improved combat maneuver feat, usually an improved trip, requires it. Usually, then they forget that they have it. And as the second level barbarian can be used to get an improved trip without the need for combat experience, many people avoid it completely that way.