Mouse click and mouse drag are out of order in Ubuntu 20.04

I recently upgraded from Ubuntu 18.04 to 20.04. Overall the interface feels much smoother. However, there is one annoying and weird issue. Mouse clicks take about 50-100 ms to register. This is imperceptible by itself. However, mouse movement does not have this same lag, so the click-move events can end up out of order. If I move my pointer to the front of some text, then I click and drag to highlight, it will miss the first few characters/lines because the OS responded to the mouse movement before it responded to the mouse click.

I first noticed this when I couldn’t reliably grab and drag a window; I kept missing the title bar on click. The best way to reproduce the effect is to open up Draw, hover near the corner, then click, then quickly move. I reliably have lines start 2-3 cm away from the corner.

This appears to only happen with my USB trackball and not the builtin trackpad or nub of my Lenovo P73, but I don’t know if that is because those devices are just a lot slower than a trackball. I tried plugging the mouse into different USB ports and always got the same behavior.

I’m guessing this is related to all the work done to make GNOME more responsive. They are probably sending mouse movement down a fast path to make it smoother. Has anyone else seen this? Does anyone have a fix?