We are starting a series of blog posts on DP-SGD that will range from gentle introductions to detailed coverage of the math and of engineering details in making it work.
我們將開始撰寫關於 DP-SGD 的一系列博客文章，內容從簡單介紹到詳細介紹數學和工程細節，以使其發揮作用。
Introduction
In this first entry, we will go over the DP-SGD algorithm focusing on introducing the reader to the core concepts, without worrying about mathematical precision or implementation just yet (they will be covered in future episodes). The intended audience for this article is someone who has experience with training ML models, including deep nets via backpropagation using their favorite framework (PyTorch, of course 🙂).
在第一篇文章中，我們將介紹 DP-SGD 算法，重點是向讀者介紹核心概念，而不用擔心數學精度或實現（它們將在以后的劇集中介紹）。 本文的目標讀者是具有訓練 ML 模型經驗的人，包括使用他們最喜歡的框架（當然是 PyTorch 🙂）通過反向傳播訓練深度網絡。
Privacy in ML
We know that training a model is an attempt at induction: we learn something from our data and we plan to use it to predict something else in the future. To state the obvious plainly, this means that there is some information in our dataset, and by training a model we condense at least some of it into an artifact we plan to use later. We learn in Machine Learning 101 that memorization can happen, so it’s perhaps not surprising that memorization can indeed be exploited to extract information about training data from a model (see eg [Carlini et al, 2018], [Feldman 2020]).
我們知道訓練模型是一種歸納嘗試：我們從數據中學習一些東西，並計划用它來預測未來的其他東西。 簡單地說，這意味着我們的數據集中有一些信息，通過訓練模型，我們至少可以將其中的一些信息壓縮成我們計划稍后使用的工件。 我們在機器學習 101 中了解到可以發生記憶，因此確實可以利用記憶從模型中提取有關訓練數據的信息（參見例如 [Carlini et al, 2018]、[Feldman 2020]），這也許並不奇怪。
What is privacy, anyway? Let’s say we don’t know and let’s start fresh by looking at our problem. We can all agree that if the ML model has never seen your data in the first place, it cannot possibly violate your privacy. Let’s call this our baseline scenario. Intuitively, if you now added your own data to the training set and the resulting model changed a lot compared to the baseline, you would be concerned about your privacy. While this makes intuitive sense, the real world is more complex than this. In particular, there are two problems we should think about:
究竟什么是隱私？ 假設我們不知道，讓我們重新審視我們的問題。 我們都同意，如果 ML 模型從一開始就從未見過您的數據，那么它就不可能侵犯您的隱私。 讓我們稱之為我們的基線場景。 直觀地說，如果您現在將自己的數據添加到訓練集中，並且生成的模型與基線相比發生了很大變化，您就會擔心自己的隱私。 雖然這很直觀，但現實世界比這更復雜。 具體來說，有兩個問題值得我們思考：
- We know that any tweak in the training process, no matter how trivial, will change the resulting model significantly. Permuting the training data, rerandomizing initial parameters, or running another task on the same GPU will produce a different model with potentially very different weights. This means that we can’t simply measure how different the weights are in these two scenarios as that will never work.
- If everyone expected that absolutely no change would happen in a model if they added their data, it means that there would be no training data and hence no ML models! We can see that this constraint is a bit too rigid.
-
我們知道訓練過程中的任何調整，無論多么微不足道，都會顯着改變結果模型。 排列訓練數據、重新隨機化初始參數或在同一 GPU 上運行另一個任務將產生具有可能非常不同的權重的不同模型。 這意味着我們不能簡單地衡量這兩種情況下的權重有多大不同，因為這永遠行不通。
如果每個人都期望在添加數據后模型中絕對不會發生任何變化，這意味着將沒有訓練數據，因此也就沒有 ML 模型！ 我們可以看到這個約束有點過於僵化了。
Luckily for us, this was figured out by [Dwork et al, 2006] and the resulting concept of differential privacy provides a solution to both problems! For the first, rather than comparing the weights of the two models, we want to consider the probabilities of observing these weights. For the second, instead of insisting that nothing will change, let’s instead promise that while something will change, we guarantee it will never change by more than a specific and predefined amount. This way, we won’t learn too much to be nosy, but we can still learn enough to produce useful models.
對我們來說幸運的是，[Dwork 等人，2006] 發現了這一點，由此產生的差分隱私概念為這兩個問題提供了解決方案！ 首先，我們不想比較兩個模型的權重，而是要考慮觀察這些權重的概率。 其次，與其堅持什么都不會改變，不如讓我們承諾，雖然某些事情會發生變化，但我們保證它的變化永遠不會超過特定和預定義的數量。 這樣，我們就不會學到太多的八卦，但我們仍然可以學到足夠的知識來產生有用的模型。
These two principles are embodied in the definition of differential privacy which goes as follows. Imagine that you have two datasets D and D′ that differ in only a single record (e.g., my data) and you interact with the data via a process or mechanism called M (this can be anything, more on this later). We can say that M is ε-differentially private if for every possible output x, the probability that this output is observed never differs by more than exp(ε) between the two scenarios (with and without my data).
這兩個原則體現在差分隱私的定義中，如下所示。 想象一下，您有兩個數據集 D 和 D'，它們僅在單個記錄（例如，我的數據）上有所不同，並且您通過稱為 M 的過程或機制與數據交互（這可以是任何內容，稍后會詳細介紹）。 我們可以說 M 是 ε-差異私有的，如果對於每個可能的輸出 x，觀察到這個輸出的概率在兩種場景（有和沒有我的數據）之間的差異永遠不會超過 exp(ε)。
Or, if you prefer a formula:
∀ D and D′ that differ in one person’s data ∀ x: ℙ[M(D) = x] ≤ exp(ε) ⋅ ℙ[M(D′) = x]
One of the amazing things about differential privacy is that it imposes no limitations on the nature on M. It can be anything. It can be a database query, it can be asking a set of questions with pen and paper to a person, or even just storing it to disk or sending it over wire, or anything else you want. As long as M enjoys this property over its outputs, then it can claim its DP badge for a specific privacy budget ε. At the same time, you can choose what ε you want to be certified for: the higher it is, the less private you are (look at the formula: it means the probabilities are allowed to diverge more). For this reason, the quantity ε is commonly referred to as the privacy [loss] budget.
差分隱私的驚人之處之一是它對 M 的性質沒有限制。它可以是任何東西。 它可以是一個數據庫查詢，它可以是用筆和紙向一個人提出一組問題，或者甚至只是將它存儲到磁盤或通過有線發送，或者任何你想要的東西。 只要 M 在其輸出上享有此屬性，那么它就可以為特定的隱私預算 ε 聲明其 DP 徽章。 同時，你可以選擇你想要認證的ε：越高，你越不私密（看公式：這意味着允許概率偏差更大）。 出於這個原因，數量 ε 通常被稱為隱私 [損失] 預算。
If we go back to our case of training a model, we now have a way to formally certify the privacy level of our training algorithm: if, after training two models, one of which on all data (mine included) and the other on all data except from mine, we can prove that all weights of the two models are observed with probabilities that lie within a predefined boundary of exp(ε) of each other, then we can claim the cool DP badge for our training procedure (that’s right! It’s the overall process that gets the badge, not the data and certainly not the trained model!).
如果我們回到訓練模型的案例，我們現在有一種方法可以正式證明我們訓練算法的隱私級別：如果在訓練兩個模型之后，其中一個針對所有數據（包括我的），另一個針對所有數據 除了我的數據，我們可以證明兩個模型的所有權重都以位於彼此 exp(ε) 的預定義邊界內的概率觀察到，然后我們可以為我們的訓練過程聲明很酷的 DP 徽章（沒錯！ 獲得徽章的是整個過程，而不是數據，當然也不是經過訓練的模型！）。
Notice that this task is harder than it looks: we can’t simply try 1000 examples (or a million, or a billion) and check whether they match. We need to prove this for all values, including never previously observed ones. The only way out of this is math and theorems. The good news about this is that if somehow we do manage to get this done, then we know that no matter what, the privacy claim is always true. There can never be any future attack that will extract our precious data from a trained module, nor any bugs to exploit to circumvent our defense just like you can’t break Pythagoras’s theorem, so this is why it’s worth doing.
請注意，這項任務比看起來更難：我們不能簡單地嘗試 1000 個示例（或一百萬或十億）並檢查它們是否匹配。 我們需要為所有值證明這一點，包括以前從未觀察到的值。 解決這個問題的唯一方法是數學和定理。 好消息是，如果我們以某種方式設法完成了這項工作，那么我們知道無論如何，隱私聲明總是正確的。 永遠不會有任何未來的攻擊會從訓練有素的模塊中提取我們寶貴的數據，也不會有任何漏洞可以利用來繞過我們的防御，就像你不能打破畢達哥拉斯定理一樣，所以這就是值得做的原因。
Providing a guarantee
So, how do we provide this guarantee then? The definition doesn’t say anything about the how.
那么，我們如何提供這種保證呢？ 該定義沒有說明如何。
It’s helpful to think about this problem on a simpler domain, so for now let us leave machine learning aside and focus on making private counting queries to a database — at the end of the day, we can see ML training as a special case of querying the training data to get numerical results out.
在更簡單的領域考慮這個問題會很有幫助，所以現在讓我們把機器學習放在一邊，專注於對數據庫進行私有計數查詢——在一天結束時，我們可以將 ML 訓練視為查詢的一個特例 訓練數據以獲得數值結果。
It is trivial to see that COUNT(*) WHERE <cond> queries can lead to a complete privacy breakdown against a sufficiently determined attacker. Consider the following example of a database that consists of two fields name and salary, with the latter being kept “private” by mandating it can only be shown in aggregates. By repeatedly running queries such as COUNT(*) WHERE name="Alice" and salary < X, Alice’s salary can be recovered with binary search. Can we defend against this attack by disallowing queries that target individuals? If only! A pair of queries COUNT(*) WHERE name<>"Alice" and salary < X and COUNT(*) WHERE salary < X get the job done just as easily as before.
很容易看到 COUNT(*) WHERE <cond> 查詢可以導致針對足夠確定的攻擊者的完整隱私破壞。 考慮以下數據庫示例，該數據庫包含兩個字段 name 和salary，后者通過強制要求只能在聚合中顯示而保持“私有”。 通過重復運行諸如 COUNT(*) WHERE name="Alice" 和salary < X 之類的查詢，Alice 的工資可以通過二分查找恢復。 我們能否通過禁止針對個人的查詢來抵御這種攻擊？ 要是！ 一對查詢 COUNT(*) WHERE name<>"Alice" 和salary < X 和 COUNT(*) WHERE sale < X 可以像以前一樣輕松地完成工作。
It may seem that these simple attacks can be thwarted by making the server’s answers a bit less precise. For instance, what if the server rounds its responses to the closest multiple of 10? Or, to confuse the attacker even more, chooses the rounding direction randomly?
A seminal result from the early 2000s due to Irit Dinur and Kobbi Nissim states, loosely, that too many accurate answers to too many questions will violate privacy almost surely. This phenomenon is known in the literature as Fundamental Law of Information Recovery and has been practically demonstrated in a variety of contexts time and time again. It effectively means that not only the answers cannot be overly precise, the error must grow with the number of answers if we want to avoid nearly total reconstruction of the dataset.
The notion of differential privacy turns these observations into actionable guidance.
The remarkable fact is that we can enforce differential privacy for counting queries by simply computing the precise answer and adding noise randomly sampled from a carefully chosen probability distribution. In its simplest form, a privacy-preserving mechanism can be implemented with a noise drawn from the Laplace distribution.
Of course, by asking the same query multiple times, the additive noise will average out and the true answer will emerge, which is exactly what Dinur-Nissim warned us about. Take that, differential privacy!
Differential privacy allows us to analyze this effect too, and in a very neat way: if you take a measurement from a mechanism with privacy budget ε₁ and a second measurement from another mechanism with privacy budget ε₂, then the total privacy budget will be simply ε₁+ε₂. Sleek, eh? This property is called (simple) composition. This means that if the mechanism guarantees that a single query has ε=1 and you want to issue three queries, the total privacy budget expended will be ε=3.
This “just add some noise” business sounds too good to be true, right? What if the attacker thinks really hard about the output of a differentially private computation, such as feeding it into a custom-made neural network trained to break privacy? Fear not! Differential privacy is preserved by post-processing, which means that results of running arbitrary computations on top of differentially private output won’t roll back the ε. Not only does it protect against clever adversaries, it gives us a lot of flexibility in designing differentially private mechanisms: once differential privacy is enforced anywhere in the data processing pipeline, the final output will satisfy differential privacy.
To recap, we learned that our solution will look like this:
- Our mechanism will be randomized, i.e., it will use noise.
- Our final privacy claim depends on the total number of interactions with the data.
- We can post-process results of a differentially private computation any way we want (as long as we don’t peek into the private dataset again).
Back to machine learning
To apply the concept of differential privacy to the original domain of machine learning, we need to land on two decisions: how we define “one person’s data” that separates D from D’ and what the mechanism M is.
Since in most applications of ML the inputs come without explicit user identifiers, with Federated Learning being one notable exception, we will default to protecting privacy of a single sample in the training dataset. We will discuss other options in future Medium posts.
As for the mechanism M, one possibility is to consider privacy of the model’s outputs only. This is indeed a valid option called private prediction, but it comes with many strings attached: the model can still memorize, so it’s up to your inference system to securely enforce those constraints. Also, this prevents us from ever releasing our ML model: if someone gets to see the weights, our privacy guarantees will be lost. This means that deploying on mobile will be considerably less safe, among others.
For this reason, it would be much preferable if we could instead insert the DP mechanism during model training, so that the resulting model could be safe for release. This brings us to the DP-SGD algorithm. (There is evidence that even when you only care about accuracy, private training still beats private prediction. See [van der Maaten, Hannun 2020] for a practical analysis and more discussion on the topic).
DP-SGD
DP-SGD (Differentially-Private Stochastic Gradient Descent) modifies the minibatch stochastic optimization process that is so popular with deep learning in order to make it differentially private.
The core idea is that training a model in PyTorch can be done through access to its parameter gradients, i.e., the gradients of the loss with respect to each parameter of your model. If this access preserves differential privacy of the training data, so does the resulting model, per the post-processing property of differential privacy.
There is also an engineering angle here: since the PyTorch optimizer is already made to look at parameter gradients, we could add this noise business directly into it and we can hide away the complexity, allowing anyone to train a differentially private model simply. Profit!
This code sample can show how simple this is
optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): x, y = batch y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Now these are filled: gradients = (p.grad for p in model.parameters()) for p in model.parameters(): # Add our differential privacy magic here p.grad += torch.normal(mean=0, std=args.sigma) # This is what optimizer.step() does p = p - args.lr * p.grad p.grad.zero_()
We have only one question left: how much noise should we be adding? Too little and we can’t respect privacy, too much and we are left with a private but useless model. This turns out to be more than a minor issue. Our ambition is to guarantee that we respect the privacy of each and every sample, not of every batch (since these aren’t a meaningful unit privacy-wise). We’ll cover the details in a future installment of this series, but the intuition is very straightforward: the right answer depends on the largest norm of the gradient in a minibatch, as that is the sample that is at most risk of exposure.
We need to add just enough noise to hide the largest possible gradient so that we can guarantee that we respect the privacy of each and every sample in that batch. To this end, we use the Gaussian mechanism that takes in two parameters, the noise multiplier and the bound on the gradient norm. But wait… The gradients that arise during training of a deep neural network are potentially unbounded. In fact, for outliers and mislabeled inputs they can be very large indeed. What gives?
If the gradients are not bounded, we’ll make them so ourselves! Let C be the target bound for the maximum gradient norm. For each sample in the batch, we compute its parameter gradient and if its norm is larger than C, we clip the gradient by scaling it down to C. Mission accomplished — all the gradients now are guaranteed to have norm bounded by C, which we naturally call the clipping threshold. Intuitively, this means that we disallow the model from learning more information than a set quantity from any given training sample, no matter how different it is from the rest.
This requires computing parameter gradients for each sample in a batch. We normally refer to them as per-sample gradients. Let’s spend a little more time here as these are a quantity that is normally not computed: usually, we process data in batches (in the code snippet above, the batch size is 32). The parameter gradients we have in p.grad are the average of the gradients for each example, which is not what we want: we want 32 different p.grad tensors, not their average into a single one.
In code:
optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): all_per_sample_gradients = [] # will have len = batch_size for sample in batch: x, y = sample y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Now p.grad for this x is filled # Need to clone it to save it per_sample_gradients = [p.grad.detach().clone() for p in model.parameters()] all_per_sample_gradients.append(per_sample_gradients) model.zero_grad() # p.grad is cumulative so we'd better reset it
Computing per-sample gradients like in the snippet above seems slow, and it is as it forces us to run backward steps for one example at a time, thus losing the benefit of parallelization. There is no standard way around this as once we look into p.grad, the per-sample information will have been already lost. It is however at least correct — a batch gradient is a per-sample gradient if batch_size=1. This method is called the microbatch method and it offers simplicity and universal compatibility (every possible layer is automatically supported) at the cost of training speed. Our library, Opacus, uses a different method that is much faster, at the cost of doing some extra engineering work. We will cover this method in-depth in a followup Medium. For now, let’s stick to microbatching.
Opacus (https://opacus.ai/) is a library that enables training PyTorch models with differential privacy
Putting it all together, we want to:
- Compute the per-sample gradients
- Clip them to a fixed maximum norm
- Aggregate them back into a single parameter gradient
- Add noise to it
Here’s some sample code to do just that:
from torch.nn.utils import clip_grad_norm_ optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): for param in model.parameters(): param.accumulated_grads = [] # Run the microbatches for sample in batch: x, y = sample y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Clip each parameter's per-sample gradient for param in model.parameters(): per_sample_grad = p.grad.detach().clone() clip_grad_norm_(per_sample_grad, max_norm=args.max_grad_norm) # in-place param.accumulated_grads.append(per_sample_grad) # Aggregate back for param in model.parameters(): param.grad = torch.stack(param.accumulated_grads, dim=0) # Now we are ready to update and add noise! for param in model.parameters(): param = param - args.lr * param.grad param += torch.normal(mean=0, std=args.noise_multiplier * args.max_grad_norm) param.grad = 0 # Reset for next iteration
This already gives a good idea of how to implement the DP-SGD algorithm, although this is clearly suboptimal and (as we shall see) not fully secure. In future Medium posts, we will cover how we bring back parallelization to DP-SGD, add support for cryptographically secure randomness, analyze the algorithm’s differential privacy, and finally train some models. Stay tuned!
To learn more about Opacus, visit opacus.ai and github.com/pytorch/opacus.
https://medium.com/pytorch/differential-privacy-series-part-1-dp-sgd-algorithm-explained-12512c3959a3
介紹
在第一篇文章中，我們將介紹 DP-SGD 算法，重點是向讀者介紹核心概念，而不用擔心數學精度或實現（它們將在以后的劇集中介紹）。本文的目標讀者是具有訓練 ML 模型經驗的人，包括使用他們最喜歡的框架（當然是 PyTorch 🙂）通過反向傳播訓練深度網絡。
機器學習中的隱私
我們知道訓練模型是一種歸納嘗試：我們從數據中學習一些東西，並計划用它來預測未來的其他東西。簡單地說，這意味着我們的數據集中有一些信息，通過訓練模型，我們至少可以將其中的一些信息壓縮成我們計划稍后使用的工件。我們在機器學習 101 中了解到記憶可能發生，因此確實可以利用記憶從模型中提取有關訓練數據的信息（參見例如[Carlini et al, 2018]和[Feldman 2020]），這也許並不奇怪。
究竟什么是隱私？假設我們不知道，讓我們重新審視我們的問題。我們都同意，如果 ML 模型從一開始就從未見過您的數據，那么它就不可能侵犯您的隱私。讓我們稱之為我們的基線場景。直觀地說，如果您現在將自己的數據添加到訓練集中，並且生成的模型與基線相比發生了很大變化，您就會擔心自己的隱私。雖然這很直觀，但現實世界比這更復雜。具體來說，有兩個問題值得我們思考：
- 我們知道，訓練過程中的任何調整，無論多么微不足道，都會顯着改變生成的模型。排列訓練數據、重新隨機化初始參數或在同一 GPU 上運行另一個任務將產生具有可能非常不同的權重的不同模型。這意味着我們不能簡單地衡量這兩種情況下權重的不同，因為這永遠不會奏效。
- 如果每個人都期望在添加數據后模型中絕對不會發生任何變化，那么這意味着將沒有訓練數據，因此也就沒有 ML 模型！我們可以看到這個約束有點過於僵化了。
對我們來說幸運的是，[Dwork et al, 2006] 發現了這一點，由此產生的差分隱私概念為這兩個問題提供了解決方案！首先，我們不想比較兩個模型的權重，而是要考慮觀察這些權重的概率。第二，與其堅持什么都不會改變，不如讓我們承諾，雖然某些事情會發生變化，但我們保證它的變化永遠不會超過特定和預定義的數量。這樣，我們就不會學到太多八卦，但我們仍然可以學到足夠的知識來產生有用的模型。
這兩個原則體現在差分隱私的定義中，如下所示。想象一下，您有兩個數據集 D 和 D'，它們僅在單個記錄（例如，我的數據）中有所不同，並且您通過稱為 M的過程或機制與數據交互（這可以是任何東西，稍后會詳細介紹）。我們可以說 M 是ε-差異私有的，如果對於每個可能的輸出x，觀察到該輸出的概率在兩種情況下（有和沒有我的數據）之間的差異永遠不會超過exp(ε )。
或者，如果您更喜歡公式：
一個人的數據不同的∀ D 和 D′ ∀ x： ℙ[M(D) = x] ≤ exp( ε ) ⋅ ℙ[M(D′) = x]
差分隱私的驚人之處之一是它對 M 的性質沒有限制。它可以是任何東西。它可以是一個數據庫查詢，它可以是用筆和紙向一個人提出一組問題，或者甚至只是將它存儲到磁盤或通過網絡發送，或者任何你想要的。只要 M 在其輸出上享有此屬性，那么它就可以為特定的隱私預算ε聲明其 DP 徽章。同時，你可以選擇你想要認證的ε：越高，你就越不私密（看公式：這意味着允許概率偏差更大）。出於這個原因，數量ε通常被稱為隱私 [損失] 預算。
如果我們回到訓練模型的案例，我們現在有一種方法可以正式證明我們訓練算法的隱私級別：如果在訓練兩個模型之后，其中一個針對所有數據（包括我的），另一個針對所有數據除了我的數據，我們可以證明兩個模型的所有權重都以位於彼此exp(ε) 的預定義邊界內的概率觀察到，然后我們可以為我們的訓練過程聲明很酷的 DP 徽章（沒錯！獲得徽章的是整個過程，而不是數據，當然也不是經過訓練的模型！）。
請注意，這項任務比看起來更難：我們不能簡單地嘗試 1000 個示例（或一百萬或十億）並檢查它們是否匹配。我們需要為所有值證明這一點，包括以前從未觀察到的值。解決這個問題的唯一方法是數學和定理。好消息是，如果我們以某種方式設法完成了這項工作，那么我們知道無論如何，隱私聲明總是正確的。永遠不會有任何未來的攻擊會從訓練有素的模塊中提取我們寶貴的數據，也不會有任何漏洞可以利用來繞過我們的防御，就像您無法打破畢達哥拉斯定理一樣，所以這就是值得這樣做的原因。
提供擔保
那么，我們如何提供這種保證呢？該定義沒有說明如何。
在更簡單的領域考慮這個問題會很有幫助，所以現在讓我們把機器學習放在一邊，專注於對數據庫進行私有計數查詢——在一天結束時，我們可以將 ML 訓練視為查詢的一個特例訓練數據以獲得數值結果。
很容易看出COUNT(*) WHERE <cond>查詢可以導致針對一個足夠堅定的攻擊者的完整隱私破壞。考慮以下由兩個字段name和組成的數據庫示例，salary后者通過強制要求保持“私有”，只能在聚合中顯示。通過重復運行查詢，例如COUNT(*) WHERE name="Alice" and salary < X，Alice 的薪水可以通過二分查找恢復。我們能否通過禁止針對個人的查詢來抵御這種攻擊？要是！一對查詢COUNT(*) WHERE name<>"Alice" and salary < X並COUNT(*) WHERE salary < X像以前一樣輕松完成工作。
似乎可以通過使服務器的答案不那么精確來阻止這些簡單的攻擊。例如，如果服務器將其響應四舍五入到最接近的 10 倍數呢？或者，為了讓攻擊者更加困惑，隨機選擇舍入方向？
由於 Irit Dinur 和 Kobbi Nissim 於 2000 年代初期的開創性結果，松散地說，對太多問題的太多准確答案幾乎肯定會侵犯隱私。這種現象在文獻中已知信息恢復的基本規律，並在各種環境中被實踐證明的時間和一次又一次。它實際上意味着不僅答案不能過於精確，如果我們想避免幾乎完全重建數據集，錯誤必須隨着答案的數量而增加。它實際上意味着不僅答案不能過於精確，如果我們想避免幾乎完全重建數據集，錯誤必須隨着答案的數量而增加。
差異隱私的概念將這些觀察結果轉化為可操作的指導。
值得注意的事實是，我們可以通過簡單地計算精確答案並添加從精心選擇的概率分布中隨機采樣的噪聲來強制計算查詢的差異隱私。在最簡單的形式中，隱私保護機制可以通過拉普拉斯分布中的噪聲來實現。
當然，通過多次詢問相同的查詢，加性噪聲會被平均化，真正的答案就會出現，這正是 Dinur-Nissim 警告我們的。拿那個，差異隱私！
差分隱私也允許我們以一種非常簡潔的方式分析這種影響：如果你從一個隱私預算為ε₁的機制中進行測量，並從另一個隱私預算為ε2 的機制中進行第二次測量，那么總隱私預算將很簡單ε₁ + ε₂。圓滑，嗯？此屬性稱為（簡單）組合。這意味着如果該機制保證單個查詢具有ε =1，並且您想要發出三個查詢，則花費的總隱私預算將為ε =3。
這種“只是增加一些噪音”的業務聽起來好得令人難以置信，對吧？如果攻擊者非常認真地考慮差分私有計算的輸出，例如將其輸入到一個定制的神經網絡訓練來破壞隱私，那該怎么辦？不要害怕！差分隱私由后處理保留，這意味着在差分隱私輸出之上運行任意計算的結果不會回滾ε。它不僅可以抵御聰明的對手，還為我們設計差分隱私機制提供了很大的靈活性：一旦在數據處理管道的任何地方強制執行差分隱私，最終輸出將滿足差分隱私。
回顧一下，我們了解到我們的解決方案將如下所示：
- 我們的機制將是隨機的，即，它將使用噪聲。
- 我們的最終隱私聲明取決於與數據交互的總數。
- 我們可以以任何我們想要的方式對差分私有計算的結果進行后處理（只要我們不再窺視私有數據集）。
回到機器學習
要將差分隱私的概念應用於機器學習的原始領域，我們需要做出兩個決定：我們如何定義將 D 與 D' 分開的“一個人的數據”以及機制 M 是什么。
由於在 ML 的大多數應用中，輸入沒有明確的用戶標識符，聯邦學習是一個明顯的例外，我們將默認保護訓練數據集中單個樣本的隱私。我們將在以后的 Medium 帖子中討論其他選項。
對於機制 M，一種可能性是僅考慮模型輸出的隱私。這確實是一個有效的選項，稱為private prediction，但它附帶了許多附加條件：模型仍然可以記憶，因此您的推理系統可以安全地強制執行這些約束。此外，這會阻止我們發布我們的 ML 模型：如果有人看到權重，我們的隱私保證將丟失。這意味着在移動設備上部署的安全性將大大降低。
出於這個原因，如果我們可以在模型訓練期間插入 DP 機制，這樣生成的模型可以安全發布，那就更好了。這將我們帶到了 DP-SGD 算法。（有證據表明，即使您只關心准確性，私人訓練仍然勝過私人預測。有關該主題的實際分析和更多討論，請參閱[van der Maaten, Hannun 2020]）。
DP-SGD
DP-SGD（Differentially-Private Stochastic Gradient Descent）修改了深度學習中非常流行的小批量隨機優化過程，以使其具有差異性私有。
核心思想是在 PyTorch 中訓練模型可以通過訪問其參數梯度來完成，即損失相對於模型每個參數的梯度。如果此訪問保留了訓練數據的差異隱私，則根據差異隱私的后處理屬性，生成的模型也會保留。
這里還有一個工程角度：由於 PyTorch 優化器已經可以查看參數梯度，我們可以直接將這個噪聲業務添加到其中，我們可以隱藏復雜性，讓任何人都可以簡單地訓練差異私有模型。利潤！
此代碼示例可以顯示這是多么簡單：
optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): x, y = batch y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Now these are filled: gradients = (p.grad for p in model.parameters()) for p in model.parameters(): # Add our differential privacy magic here p.grad += torch.normal(mean=0, std=args.sigma) # This is what optimizer.step() does p = p - args.lr * p.grad p.grad.zero_()
我們只剩下一個問題：我們應該添加多少噪音？太少了，我們就不能尊重隱私，太多了，我們就會留下一個私人但無用的模型。事實證明，這不僅僅是一個小問題。我們的目標是保證我們尊重每個樣本的隱私，而不是每個批次的隱私（因為這些在隱私方面並不是一個有意義的單位）。我們將在本系列的后續部分中詳細介紹，但直覺非常簡單：正確答案取決於小批量中梯度的最大范數，因為這是暴露風險最大的樣本。
我們需要添加剛好足夠的噪聲來隱藏最大可能的梯度，以便我們可以保證我們尊重該批次中每個樣本的隱私。為此，我們使用接受兩個參數的高斯機制，噪聲乘數和梯度范數的界限。但是等等……在深度神經網絡的訓練過程中出現的梯度可能是無限的。事實上，對於異常值和錯誤標記的輸入，它們確實可能非常大。是什么賦予了？
如果梯度沒有界限，我們將自己制作它們！令 C 為最大梯度范數的目標界限。對於批次中的每個樣本，我們計算其參數梯度，如果其范數大於 C，我們通過將其縮小到 C 來裁剪梯度。自然叫剪裁閾值。直觀地說，這意味着我們不允許模型從任何給定的訓練樣本中學習比一組數量更多的信息，無論它與其他樣本有多大不同。
這需要為批次中的每個樣本計算參數梯度。我們通常將它們稱為每樣本梯度。讓我們在這里多花一點時間，因為這些是通常不會計算的數量：通常，我們分批處理數據（在上面的代碼片段中，批大小為 32）。我們的參數梯度p.grad是每個示例的梯度的平均值，這不是我們想要的：我們想要 32 個不同的p.grad張量，而不是將它們的平均值合並為一個。
在代碼中：
optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): all_per_sample_gradients = [] # will have len = batch_size for sample in batch: x, y = sample y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Now p.grad for this x is filled # Need to clone it to save it per_sample_gradients = [p.grad.detach().clone() for p in model.parameters()] all_per_sample_gradients.append(per_sample_gradients) model.zero_grad() # p.grad is cumulative so we'd better reset it
像上面的代碼片段那樣計算每個樣本的梯度似乎很慢，因為它迫使我們一次為一個示例運行后退步驟，從而失去了並行化的好處。沒有標准的方法可以解決這個問題，因為一旦我們調查p.grad，每個樣本的信息將已經丟失。然而，它至少是正確的——如果 ，批處理梯度是每個樣本的梯度batch_size=1。這種方法稱為微批處理方法，它以訓練速度為代價提供簡單性和通用兼容性（自動支持每個可能的層）。我們的圖書館，Opacus, 使用一種更快的不同方法，代價是做一些額外的工程工作。我們將在后續的 Medium 中深入介紹這種方法。現在，讓我們堅持使用微批處理。
Opacus ( https://opacus.ai/ ) 是一個能夠訓練具有差異隱私的 PyTorch 模型的庫
綜上所述，我們希望：
- 計算每個樣本的梯度
- 將它們剪輯到固定的最大規范
- 將它們聚合回單個參數梯度
- 給它添加噪音
下面是一些示例代碼來做到這一點：
from torch.nn.utils import clip_grad_norm_ optimizer = torch.optim.SGD(lr=args.lr) for batch in Dataloader(train_dataset, batch_size=32): for param in model.parameters(): param.accumulated_grads = [] # Run the microbatches for sample in batch: x, y = sample y_hat = model(x) loss = criterion(y_hat, y) loss.backward() # Clip each parameter's per-sample gradient for param in model.parameters(): per_sample_grad = p.grad.detach().clone() clip_grad_norm_(per_sample_grad, max_norm=args.max_grad_norm) # in-place param.accumulated_grads.append(per_sample_grad) # Aggregate back for param in model.parameters(): param.grad = torch.stack(param.accumulated_grads, dim=0) # Now we are ready to update and add noise! for param in model.parameters(): param = param - args.lr * param.grad param += torch.normal(mean=0, std=args.noise_multiplier * args.max_grad_norm) param.grad = 0 # Reset for next iteration
這已經給出了如何實現 DP-SGD 算法的好主意，盡管這顯然是次優的並且（正如我們將看到的）不完全安全。在以后的 Medium 帖子中，我們將介紹如何將並行化帶回 DP-SGD，添加對加密安全隨機性的支持，分析算法的差分隱私，並最終訓練一些模型。敬請關注！
要了解有關 Opacus 的更多信息，請訪問opacus.ai和github.com/pytorch/opacus。