Security News > 2022 > May > Manipulating Machine-Learning Systems through the Order of the Training Data
Most deep neural networks are trained by stochastic gradient descent.
Now "Stochastic" is a fancy Greek word for "Random"; it means that the training data are fed into the model in random order.
Suppose for example a company or a country wanted to have a credit-scoring system that's secretly sexist, but still be able to pretend that its training was actually fair.
Well, they could assemble a set of financial data that was representative of the whole population, but start the model's training on ten rich men and ten poor women drawn from that set then let initialisation bias do the rest of the work.
Previously, people had assumed that in order to poison a model or introduce backdoors, you needed to add adversarial samples to the training data.
If an adversary can manipulate the order in which batches of training data are presented to the model, they can undermine both its integrity and its availability.