That’s according to this Tweet from Crawford that specifies that since Magical Secrets spells count as Bard spells, you can trade them out when you level up, but only for a spell on the Bard list. Personally however, I think this is a pretty vaguely worded ruling and doesn’t really answer the user’s question.
That being said, this is a very unfun ruling and this question isn’t tagged RAW.
Magical Secrets is one of the most awesome things about Bards, and makes them tremendously powerful. Being able to take any spell from the book means you can do things no other class can do, like grant your high DPR party member an extra attack using Haste while also preventing some terrible spells being cast on them using Counterspell (enhanced even further using Jack of All Trades). Or taking Wish at 18th level and making your DM hate you.
At the same time though, having all the good spells from every class makes the rest of the party pretty jealous. It’s bad enough for the PHB Ranger to not get 5th level spells until 17th level, it’s even worse when the party Bard can take all but two of your spells you choose the next time you level up.
I’ve found that the most optimal solution is this: if a Bard wants to replace a spell they learn via Magical Secrets at a lower level, they can only choose a new spell that they would have been able to learn at the level they were at when they chose it.
So if a Lore Bard chose Counterspell and Booming Blade at 6th level using Lore Bard’s Additional Magic Secrets feature but later learned Booming Blade through the Magic Initiate feat at 8th level, they would only be able to replace Booming Blade with a spell of 3rd level or lower, because that’s the max level they could learn at Level 6. (Shield is a good choice for this by the way)