In a recent obituary of distinguished music critic Andrew Porter, I was struck by and agree with his sentence describing a well-known conductor's antics on the podium: … if one averts one's eyes one hears something less extraordinary than the performance he mimes.
In my own occasional efforts at conducting, I am well aware of this danger and find it usually preferable to direct whilst playing the violin which is preferable also for the musicians being directed!