Federated learning in the fight against e-mail threats

What is the easiest way to find a threat (either phishing or spam) in your e-mail? A variety of technical headers and other indirect markers of an unwanted message can point the way, but we shouldn’t forget the most obvious bit — the message text. One might think it’s the first thing to analyze; after all, the text is what cybercriminals or unscrupulous advertisers use to manipulate recipients. The task isn’t quite that simple, though; whereas signature analysis could cope with the task in the past, it is now necessary to analyze the text using machine-learning algorithms. And if the machine learning model is to be trained to classify messages correctly, it needs to be fed messages in significant quantities — and that is not always practical, for privacy reasons. We found a solution.

Why isn’t signature analysis effective anymore?

Ten years ago, catching a huge proportion of unwanted e-mail based purely on message text was relatively easy because cybercriminals used the same templates — the text of spam (and phishing) messages hardly changed. Today, cybercriminals continually improve the efficiency of their mailings, and they use millions of hooks: new video games, TV series, or smartphone models; political news; even emergencies (take, for example, the abundance of phishing and spam related to COVID-19). The massive variety of topics complicates the detection process. Moreover, attackers can even vary the text within one mailing wave to elude e-mail filters.

Of course, signature-based approaches are still in use, though their success basically relies on encountering text that someone has already classified as unwanted or harmful. They can’t work proactively because spammers can bypass them by making changes to mailing text. The only way to deal with this problem is through machine learning.

What’s the problem with learning?

In recent years, machine-learning methods have shown good results in solving many problems. By analyzing a large amount of data, models learn to make decisions and find nontrivial common features in an information stream.  We use neural networks trained on technical e-mail headers, together with DMARC, to detect e-mail threats. So, why can’t we just do the same thing with message text?

As mentioned above, models need a huge amount of data. In this case, the data consists of e-mails, and not only malicious ones — we need legitimate messages as well. Without them, teaching the model to distinguish an attack from legitimate correspondence would be impossible. We have numerous e-mail traps that catch all sorts of unwanted e-mails (we use them to make signatures) but obtaining legitimate letters for learning is a more complicated task.

Typically, data is collected on servers for centralized learning. But when we are talking about text, additional difficulties arise: E-mails can contain private data, so storing and processing them in their original form would be unacceptable. So, how can we obtain a large enough collection of legitimate e-mails?

Federated learning

We solve that problem by using the federated learning method, neatly eliminating the need to collect legitimate e-mails and instead training models in a decentralized way. Model training takes place directly on the client’s mail servers, and the central server receives only the trained weights of the machine-learning models, not message text. At the central server, algorithms combine the data with the resulting version of the model, and then we send it back to client’s solutions, where model again proceeds to analyze the stream of e-mails.

That’s a slightly simplified picture: Before the newly trained model is set loose on real letters, it goes through several iterations of additional training. In other words, two models work simultaneously on the e-mail server: one in training mode, the other in active mode. After several trips to the central server, the retrained model replaces the active one.

It’s impossible to recover the text of specific e-mails from the model weights; thus its privacy during processing is assured. Nevertheless, training on real e-mails significantly improves the detection model’s quality.

At the moment, we are already using this approach to spam classification, in test mode, in Kaspersky Security for Microsoft Office 365, and it’s showing outstanding results. Soon, it will be applied more widely and used to identify other threats such as phishing, BEC, and more.