Knowing where an invisible creature is allows you to attack it at a disadvantage.
There are two related questions, here and here, that explain that if you are invisible, other creatures still know where you are because they can still hear you. The "Invisible attacks" section says:
When you attack a target that you cannot see, you have a disadvantage in the attack roll. This is true if you are guessing the location of the target or if pointing to a creature that you can hear but not see…
Therefore, if the target is simply invisible, but can still be heard, it would already know its location and could attack it at a disadvantage as described above.
Keep in mind that even if a creature were hidden (invisible and inaudible) it could still guess its location, however, the section quoted above continues, saying:
If the target is not in the location you pointed to, it automatically fails, but the DM generally only says that the attack failed, not if you guessed the location of the target correctly …
If, on the other hand, a creature is hidden and you bump into it, you would presumably know where it is and, therefore, could make an attack at a disadvantage, as in the invisible case. However, there is at least one strange problem that results from the "Move around other creatures" rule:
You can move through the space of a non-hostile creature. On the contrary, you can move through the space of a hostile creature only if the creature is at least two sizes larger or smaller than you. Remember that another creature's space is difficult terrain for you.
If an enemy were a size larger or smaller than you, then you would be prevented from moving into your space and could simply attack that space at a disadvantage.
However, if an enemy were at least two sizes larger or smaller than you, then he would enter his space and collide with him and, therefore, would supposedly know where he is. But you can't attack them, this is due to the following rule:
Whether a creature is a friend or an enemy, you cannot voluntarily end your movement in its space …
As shown in this Q / A, the above quotation applies to all your movement, which means you cannot make an attack while in another creature's space if you know they are there. Therefore, you should first leave the creature's space to attack it.
A possible exception is if you were out of motion when you entered the creature space. You didn't know they were there, so this would not be a voluntary end of your movement there, it just happened that a creature was there when you ran out of movement. Therefore, (probably) you will be allowed to attack even while in your space.
I asked this Q / A because I was not sure of the exact decision in this case.
You can attack an invisible creature at a disadvantage if you know where they are, otherwise you would have to guess its location.