I know this might be unpopular, but, I don't know, you could just QUIT the app? Obviously, there's a bigger issue here if Zoom is listening in when we don't want them to. But, the number of comments providing workarounds just leaves me thinking: why don't you just quit the app? Immediate problem solved. Long-term problem not solved.
Zoom is not just simply a meeting app, it has chat function and integrated apps too. People who don’t use Zoom just for meeting purpose might find the situation uncomfortable.
The issue is that Zoom continues to keep the mic open after you quit the App. They blamed it on a bug and said it was fixed in a new version but it's still happening! This is why doing this in the browser would be better or even adding an App Firewall like OverSight to block access to the external devices.