All speakers are set to react when you use the keywords to interact with Google. To eliminate multiple speakers talking at the same time, using the "Microphone Mute" button will stop the speaker from reacting to the keywords.