greybus: audio: Fix incorrect counting of 'ida'

Function gb_audio_manager_remove_all() to remove all audio modules,
doesn't control correctly 'ida' counting.

Signed-off-by: Dinko Mironov <dmironov@mm-sol.com>
Signed-off-by: Greg Kroah-Hartman <gregkh@google.com>