Who is who?
Klang has always had the ability to separate different voices in a meeting into different speakers. Those speakers are numbered Speaker 1, Speaker 2, and so on. Sometimes we can even infer the names of those speakers based on participant names in a meeting. However, there has been no way to identify people between meetings. Is "Anna" in one meeting the same person as Anna in another meeting? And how do we know if a voice in an uploaded file is that same person? This is the problem studied at Klang. We wanted to solve it because it would automatically lead to several improvements. For example, summaries could list who said what, you could search for meetings by a specific speaker, and even figure out what YOU said in a conversation. In this first of two posts we will describe how we approached this problem, what challenges we faced, and what results we got.
Recognizing voices
The standard approach for recognizing voices is to turn each clip of speech into a speaker embedding: a list of numbers that represents how a voice sounds. There are several open models that do this, and they are all trained to achieve one goal: clips of the same person should end up with similar numbers, i.e. "close" to each other, and clips of different people should end up far from each other. This means that the models inherently try to ignore everything about what is being said, and represent only the properties of the voice itself.
A note on security and privacy
The fact that the models are trained to ignore what is being said is important for security and privacy. While text embeddings used for search have been shown to contain a lot of information about the content of the text, speaker embeddings are designed to be content-agnostic from the start. This means that saving these embeddings is a lot safer than saving the audio or the transcript itself. The numbers in an embedding represent what a person sounds like, which is far less sensitive than the words themselves. With this said, all speaker embeddings in Klang are stored encrypted, isolated to each organization. It is also possible to disable speaker embeddings entirely if an organization does not want to use them.
Testing on internal data
A core principle at Klang is that we do not train models on our customers' data. So we test our approach on internal and public data instead. Luckily for us, we have several years of recording all of our internal meetings in Klang! We extracted tens of thousands of clips from these meetings and computed an embedding for each. Below is a sample of these embeddings projected down to two dimensions. Every dot is one clip, every color one colleague.
It's clear that the model is able to separate the voices of different people, even though it has never heard any of our voices before.
How much does a voice vary?
The clusters in the plot are encouraging, but we still need to know how well it works. There are many factors that can change how a person sounds, such as the microphone, the room, the volume level, or even a cold. And even within the same recording, the distance to the microphone or the background noise can affect the audio. We need to know how much these factors influence the embeddings, and we need to know how big this variation is compared to the variation between different people.
We used the internal meetings described above to measure the variation in speaker embeddings. We use cosine similarity, i.e. 1.0 means exactly the same, and 0.0 means completely different. The data confirms what one would expect: two clips from the same person and the same meeting are very similar (0.86), the same person in different conversations is less similar (0.76), and two different people are quite different (0.26). However, these numbers are averages, and vary a lot from clip to clip. This means that no matter where we put a cutoff, e.g. 0.5, some clips from the same person will be below that cutoff, and some clips from different people will be above it.
We therefore had to study exactly how embeddings vary, not just how much. We compiled a dataset of almost 8,000 different voices from publicly available sources and computed embeddings for each. This revealed that embeddings vary from each other in a very uneven way in the embedding space. In some directions the embeddings are very far apart, and in other directions they are close together. This is a large part of what made it so hard to find a single cutoff that works for all voices. We therefore computed a warping of the embedding space that makes the embeddings more evenly distributed. In a way, this spreads the voices out more evenly in the space, allowing more distance between similar voices.
This actually decreased the similarity between embeddings from different people, from 0.26 to 0.08. This is enough to be able to almost always separate different voices. This leaves a few questions for the next part of this article: How often is it ok to misidentify a voice? And is there more we can do to make that even more unlikely?
Read the second part of this article here: Unlikely similar voices