The navigation around a busy canvas is awkward. For example having to switch from Select mode to Grab mode to move around is not intuitive. Relatedly, when in grab mode and you hover over a chart for example, you would expect to be able to click the chart and then interact with it. You cannot. You must switch to select mode to do that.
I suggest that if the mouse is in “white space” then grab mode is in effect by default. If the mouse is over an object then it should switch to select mode automatically.