Using the User Action Driven Scroll can help
If I understand it correctly, it has a global displacement (applied to the entire page) and another displacement within a particular card. Now, while moving vertically, the nested displacement is causing irritation, because the behavior goes against the action / mental model predicted by the user.
One approach you can try is this:
Instead of showing a scroll by default, it only enables it if the user clicks the "See more / load more" button.
As soon as the user clicks the see more / load more button, the scroll is activated
Now, it can be active until the user clicks somewhere outside the box or click on any of the other boxes.
As soon as the user clicks on any of the other boxes, the local movement of the card will be activated and the displacement of the previous box will disappear:
If the user clicks anywhere else, and not on any of the cards, it returns to state 1, and now you can use the global scroll to move around the board. The global offset is also applicable if the user's mouse is not placed inside the box that has a local offset.