This is a good mental exercise for learning about DND combat encounters…that’s for sure.
I don’t know the “actual answer”, except maybe “If you defeated the spell caster, then no”. But, if you’re reading this looking for a path toward a practical solution in the case where the party does not definitely defeat the spell-casting enemy in combat…..analyze the Treant and the Galeb Duhr. See what comparisons you can make and what insights you can glean….
A Treant is roughly a CR 6 without it’s Animate Tree (“AT”) action, but in actual fact it has a CR of 9 due in large part to the max. damage/round boost AT gives it (or, AT applies an XP Encounter Multiplier of 1.5-2x, as you go from facing 1 monster to up to 3 monsters).
Galeb Duhr is another monster we can apply the same thought process to, which as an action can cast Animate Boulders (“AB”). Without AB, the Galeb Duhr has a CR of 3 or 4. Again, because of the jump in max. damage per round, it’s a CR of 6 (or, AB applies an XP Encounter Multiplier of 2x, as you go from facing 1 monster to 3 monsters). Though I believe this is a concentration spell for the Galeb Duhr (it’s not for the Treant).
Thoughts:
For a party of 13th level adventurers, combat against a Treant without AT is of “medium” Encounter Difficulty (this is all “in my estimation”, looking at the DMG pgs 82 & 274-5). However, with AT, the Encounter Difficulty is downright “deadly”. So, instead of 2,300 XP as a CR 6 without AT…. with AT, as a CR 9, it imparts >2x the XP (5,000 XP).
Key questions
Without Animate Objects, how much damage is the enemy spell-caster able to inflict? What’s its best damage concentration spell, otherwise?
If you killed the 2 Animated Trees, but failed to kill the main spell-casting Treant, would your adventurers gain any XP? If so, how much? Why?
If the Treant was somehow unable to cast Animate Tree effectively (no trees within 60 feet, say) would you still get the full 5,000 XP for the kill? Likely yes.
If the Treant was an ally NPC, and assisting the party in battle but was unable or refused to use AT, would the DM still give the Treant the same cut of XP gained from the encounter compared to if it had used AT?
*What does putting another 3-10 enemy targets (@ 5th lvl), or adding an additional 10d4+40 max damage/round *(as in the case of animating 10 “Tiny” objects that do 1d4+4 damage each…with ACs of 19 & +8 to hit, a fly speed of 30 and darkvision….) do for the enemy?
How dangerous is the enemy without that ability, in the context of the combat encounter (the enemy caster could be 500 feet away, still giving commands to the constructs…!)
Or, put another way, what’s the CR of that particular enemy without Animate Objects?
&
For the particular combat Encounter, how significant is the impact of Animate Objects on the total Encounter Difficulty?
Final Thoughts
If you kill or successfully defeat the spell-casting enemy, I don’t see a legitimate reason to count XP from defeating the animated object constructs.
However, if the party makeup is such that it faces a significant degree of difficulty and successfully vanquishes the animated object creatures, and accomplishes some goal/side-quest — even despite failing to kill or otherwise defeat the enemy spell-caster for whatever reason — then I think XP should be awarded and tied to the combat.
If you have 11 enemies on the battlefield (Spell caster + 10 constructs), that’s an XP Encounter Multiplier of 3. If you figure each one is worth even just 10 XP….10x10x3 = 300xp? Sounds good for a 3rd-4th level party who needs to extract a valuable item from a castle guarded by 10 “tiny” animate object constructs, but clearly isn’t @ a high enough level to defeat the evil wizard controlling the constructs.
Maybe they escape by the skin of their teeth with the treasure, after defeating the constructs in a 10 round combat session, just before the wizard arrives in person to destroy them (but was several hundred feet away, in another part of the castle). Nail biter, worthy of combat XP.
Again….
This is a good mental exercise for learning about DND combat encounters…that’s for sure.