CR is an out-of-universe concept, and your in-universe character does not know whether a creature is above or below the required CR. As far as I’m aware your character can select invalid targets for spells, and they will simply not work if they are not correctly targeted.
For an analogous example, if a spell requires the target to be willing, and you cast it on a target that is not actually willing, you still cast the spell but it has no effect. That will use your turn and your spell slot since your character didn’t know whether the spell was going to work until after they cast it.
In-universe in this situation your character would only know that the creature was unaffected, and would not know whether that was due to the creature’s CR or their Will save since either of those would result in the same lack of effect. If I were DM’ing that situation I would only tell the player that their spell didn’t succeed, and not that the creature’s CR was too high, but that is definitely up to the style and discretion of your DM.
One way to avoid this sort of situation would be to roll a knowledge check on your opponent before fighting them. If you roll high enough the DM could inform you (the player, not the character) of that creature’s CR as well as some of their other traits. The in-universe equivalent of that would be the knowledge of the creature’s relative strength, which would tell your Mystic that they shouldn’t waste their time and spell slots casting Fear on them.
The OP found a citation to support this answer. From the core rule book section on “Casting Spells”, at the end of the sub-section “Concentration and Interrupted Spells” it says:
If you ever try to cast a spell in conditions where the
characteristics of the spell can’t be made to conform, the spell
fails. For example, if you try to cast a spell that targets a humanoid
on a non-humanoid, the spell fails.
This would easily be extended to the CR example as the character used a spell where the conditions could not be made to conform to the requirements of the spell