Collaborative Computing for Linear Regression with Privacy
Albert Guana,1 , Po-Wen Chia,2,, Chun-Hung Richard Linb,3
a
Department of Computer Science and Information Engineering, National Taiwan Normal
University, Taipei, 11677, Taiwan
b
Department of Computer Science and Engineering, National Sun Yat-sen
University, Kaoshiung, 80424, Taiwan
Abstract
Machine learning usually requires a large number of training data to build a useful model.
We exploit the mathematical structure of linear regression to develop a secure and privacypreserving method that allows multiple parties to collaboratively compute optimal model
parameters without requiring parties to share their raw data. The new approach also
allows to efficiently delete the data of users who wants to leave the group and wish to
have their data deleted. Since the data will remain private during both learning and
unlearning, data owners are more willing to contribute their collected datasets to build
better models, benefiting all parties. The proposed collaborative computation of linear
regression model does not require a trusted third party, thereby avoiding the difficulty of
building a robust trust system in the current Internet environment. The proposed scheme
is also computationally efficient and does not use encryption to keep data secret. We also
show that the new linear regression learning scheme can be updated incrementally, which
means that new datasets can be easily incorporated into the system, and some data can
be removed from the system to obtain a more suitable linear regression model without
recomputing from scratch.
Keywords: Collaborative learning, privacy protection, incremental computation,
machine unlearning.
1. Introduction
Machine learning attempts to build models based on training data to make predictions
or decisions without being explicitly programmed. It usually requires a large number of
training data to build a useful model. In many practical applications, these data may not
belong to one organization, but rather be distributed across multiple sites. For example,
every hospital collects medical data of its patients. The data collected in one hospital is
often insufficient to obtain a good model for developing new drugs or treatments for some
specific disease. Therefore, sharing data is very important in machine learning when the
data is distributed across many sites.
Like medical data, almost all of the data contain private and sensitive personal information. When sharing these data, security or personal privacy concerns must be properly
Email addresses: albert.zj.guan@gmail.com (Albert Guan), neokent@gapps.ntnu.edu.tw
(Po-Wen Chi), lin@cse.nsysu.edu.tw (Chun-Hung Richard Lin)
1
The research of the first author is supported in part by the MOST project 107-2218-E-110-017-MY3.
2
The research of the second author is supported in part by the MOST project 107-2218-E-110-017MY3.
3
The research of the third author is supported in part by the MOST project 108-2622-E-110-009-CC3.
Preprint submitted to Information Processing Letters
July 21, 2023
addressed. The release of personal data in almost all countries in the world is restricted
by laws and regulations. For example, the European Data Protection Regulation is applicable in 2018 for all member states to harmonize data privacy laws across European
Union [1]. Another reason for limiting data sharing is that collecting these data usually
requires a lot of time and effort. Data owners usually view the data they collect as a
valuable asset. They will be more willing to share the collected data with others only if
it is legal and mutually beneficial.
In this article, we consider the security and privacy-preserving of a specific machine
learning problem: linear regression. Linear regression models the relationship between
variables by fitting a line to the observed data [2, 3]. It is the most basic model in
machine learning and has many applications [4]. We propose a method to collaboratively
build a linear regression model without releasing the original data. The new learning
scheme also supports deleting a user’s dataset without using the user’s original data.
Therefore, it complies with privacy protection regulations.
Secure collaborative machine learning is also a type of secure multiparty computation, which has been widely studied by cryptographers. In secure multiparty computing,
no data can be sent directly to the other parties for security reasons. There are many
techniques to achieve secrecy in multiparty computation. General method for secure multiparty computation, such as Yao’s garbled circuits, may not be efficiently implemented.
By exploiting the mathematical structure of linear regression, we achieve efficient secure
construction of linear regression models.
In secure computation, the goal is to provide a well-defined framework to guarantee
complete zero knowledge about other party’s private data. Zero knowledge is very desirable, but this desired property usually requires heavy computations [5]. Furthermore,
zero-knowledge only holds for polynomial time attackers, i.e., these schemes can only be
computationally secure. In our scheme, we show that the curious party does not have
enough information to recover the exact value of the other party’s original data.
It may be useful to design a scheme with a trusted third party, especially when the
third party only needs to play a light role in the protocol. However, in the current Internet
environment, it is often difficult to establish and maintain long-term trust among parties,
especially when the number of participants increases. Such a system may not be very
robust; as long as the server or some party violates the security rules, the system cannot
function properly. The proposed method for collaboratively constructing a linear model
in this article does not require trusted third parties. We only assume that all parties are
semi-honest: they all follow the protocol, but they are also curious about the data of the
other parties.
Encryption can be used to protect sensitive information. However, the data in machine
learning must be used for efficient computation, and general encryption schemes do not
support this feature. Fully homomorphic encryption, such as the Paillier cryptosystem [6],
may be used in computing model parameters in machine learning. However, homomorphic
encryption usually requires intensive computation to achieve a required level of security,
and these cryptosystems can only achieve computational security.
Most importantly, numbers in machine learning are usually real numbers. These
numbers must be represented in a standard format (such as IEEE Standard 754 floatingpoint numbers) so that currently available computer CPUs can perform the calculations.
On the other hand, the plaintext and the ciphertext of a cryptosystem are usually finite
groups, finite rings, or fields to avoid rounding errors. Almost all these cryptosystems,
even if they are fully homomorphic, may not be suitable for machine learning. Recently,
2
Cheon et al. developed the CKKS cryptosystem [7] to support a special type of fixedpoint arithmetic, called block floating-point arithmetic. Although floating-point numbers
can be approximated in fixed-size memory, current version of CKKS support only a limited
number of multiplications before a time-consuming bootstrapping step is taken. Another
limitation of CKKS is that it currently does not directly support the floating-point division
operations required for linear regression and almost all machine learning models. In this
regard, Barbenko et al. proposed a two-stage method for Euclidean division with the
CKKS scheme and studied its properties [8]. In this article, we propose a new scheme
to securely compute optimal parameters for linear regression without encryption to avoid
all these problems. The main contributions of this research are summarized as follows.
1. The proposed secure and privacy protection machine learning scheme for linear
regression does not need trusted servers or encryption.
2. Its security is not based on computationally hard problems. Even an attacker with
unlimited computing power does not have enough information to learn the exact
value of the private data of the other parties.
3. It is computationally efficient. It does not require encryption and does not incur
additional computation due to distribution of the datasets.
4. The proposed scheme only sends aggregated data to other parties. The size of the
aggregated data is proportional to the number of features. In practical applications,
whether encrypted or not, the amount of data is usually much larger than the
number of features. This implies that the communication cost of our scheme is
lower than sending raw data, whether they are encrypted or not.
5. The new linear regression learning scheme can be updated incrementally. New
datasets can be easily incorporated into the system and old dataset can be removed
from the system to obtain a more suitable linear regression model without recomputing the model parameters from scratch.
2. Related Work
In contrast to traditional centralized machine learning techniques where all local
datasets are uploaded to one server, the technique we study is known as federated learning or collaborative learning. In federated learning, the most important task is to devise
a method for parties to collaboratively compute the parameters of a machine learning
model from their input data, while keeping their input data private. To meet the security
requirement, trusted servers can be used to protect the confidentiality of data and perform computations efficiently. The security of many known secure computation schemes
for linear regression requires servers [9, 10, 11, 12, 13, 14]. Some of the schemes use only
one server [9, 12, 13]. some of the schemes use two servers, one for cryptography service
and the other for computation [10, 11, 14].
There are different ways of using the servers. In Cock’s et al. scheme [12], a trusted
server is only needed in the pre-distribution phase to do initialization. On the other
hand, in Chen et al. [9] and Mandal et al.’s schemes [13], the server is required to do
computational tasks, but it only required the server to be semi-honest.
Federated learning can also be considered as secure multiparty computation, or privacypreserving computation. Secure multiparty computation has been widely studied by many
cryptographers [15, 16, 17]. Recently, Ma et al. developed algorithms under secure multiparty computation which enable pharmaceutical institutions to achieve high-quality models to advance drug discovery without divulging private drug-related information [18].
3
The security of most existing secure multiparty computations depends on computationally hard problems, such as factoring large integers or solving discrete logarithms in
large groups. Some of the schemes based on Yao’s garble circuit and combined with secure
computation for computing inner product [10]. Some of the schemes based on homomorphic encryption to ensure the security of the data [11]. Kikuchi et al. proposed a scheme
that does not require trusted servers [19], but it need Paillier’s cryptosystem, which is
a public key cryptosystem, as well as homomorphic encryption. These cryptosystems,
including Yao’s garbled circuits, are computationally intensive.
In our proposed scheme for linear regression, with proper bookkeeping, datasets owned
by a users can also be deleted from the computations efficiently. This is useful when a
party leaves the group and wants to remove information about its party. This is called
machine unlearning. Machine unlearning is necessary to comply with privacy protection
laws and regulations, such as GDPR in European countries. In our new scheme, both
the addition and deletion of some datasets can be done efficiently without recomputing
the model parameters from scratch. However, our scheme needs to invert a matrix of size
(m + 1) × (m + 1), where m is the number of features used in the linear regression model.
This is not a problem in practice since the value of m is usually small compared to the
amount of data n.
Some of the known schemes do not use the closed form of the optimum solution for
linear regression [10, 13]. Although the authors claimed that the advantage is that their
scheme does not need to compute the matrix inverse, the drawback is that the result will
only be an approximation. Most importantly, this type of computation method has to send
gradients descent, or other related data, multiple times to make the model parameters
accurate enough. It is well known that sending gradient descent iteratively can reveal
enough information to the attacker to infer the value of the training data [20]. This is
because the attacker can accumulate some information from each gradient descent. When
enough information is collected, the attacker can deduce the values of the original training
dataset.
For each dataset, our scheme only sends the aggregated data once, and we show that
the information leaked is too small for any unauthorized party to infer information about
the other party’s data. Note that this technique works well for linear regression problems.
Other federated learning problems may not be able to avoid iterative send gradient descent
and require other techniques to preserve privacy. For example, in Deep Neural Networks
(DNN), Xu et al. proposed a double masking protocol to guarantee the confidentiality of
user local gradients during federated learning [21].
3. The Linear Regression Problem
In this section, we first briefly describe the computational procedure for linear regression. We then show a method to compute the optimal parameter of a linear model by
minimizing the sum of the square of errors. This classical method is called the least square
method [22, 4].
Linear regression is a linear approach to modeling the relationship between a response
y and a set of m independent variables x1 , x2 , . . . , xm in an experiment. It can be expressed
as
y = β0 + β1 x1 + · · · + βm xm .
These independent variables x1 , x2 , . . . , xm are also called the features of the model.
4
Suppose n experiments have been performed, each with different setting of the features
x1 , x2 , . . . , xm . With a set of n observed data, we obtain n equations.
yi = β0 + β1 xi1 + β2 xi2 + · · · + βm xim ,
i = 1, 2, . . . , n,
where xij is the value of variable xj used in the i-th experiment. It is convenient to write
these n equations in matrix form. Let ⃗xi = [xi1 , xi2 , . . . , xim ], and let the model coefficients
P
i
be: β⃗ = [β0 , β1 , . . . , βm ]. The model prediction would be yi = β0 + m
xi
j=1 βj xj . If each ⃗
is extended to ⃗xi = [1, xi1 , xi2 , . . . , xim ], then
yi =
m
X
⃗ xi .
βj xij = β⃗
j=0
In constructing a linear model, the coefficients β0 , β1 , . . . , βm are unknowns. A set of
n experiments can be performed to determine possible values of these m + 1 unknowns.
These m + 1 unknowns β0 , β1 , . . . , βm are called parameters of the linear regression model.
If all the experiments have no errors and the setting of the variables ⃗xi are independent,
then m + 1 experiments are sufficient to determine the m + 1 unknowns β0 , β1 , . . . , βm .
However, almost all experiments cannot avoid errors. It needs more than m + 1 experiments, usually much more experiments, to obtain “optimal” values of β0 , β1 , . . . , βm .
To define the optimal values for these parameters, we first define the loss function.
Given a set of values for β⃗ = [β0 , β1 , . . . , βm ], we can compute the loss with respect to β⃗
as
n 2
X
⃗ xi − yi .
⃗ =
β⃗
L(β)
i=1
In the least square setting, the optimum parameter is defined as the one that minimizes
the sum of mean square loss:
n 2
X
⃗
⃗ xi − yi .
β⃗
β̂ = arg min L β⃗ = arg min
i=1
4. The Proposed Collaborative Learning Method
In this section, we first describe the theoretical basis on which our proposed collaborative learning approach is based. We then describe new collaborative learning methods.
4.1. Theoretical Basis of the Proposed Scheme
First, we show how to compute the optimal parameters of a linear regression model
incrementally. Assume that the number of features is 2. Thus,
y = β0 + β1 x1 + β2 x2 .
The optimal values of each βj , j = 0, 1, 2, can be obtained by finding the minimum value
⃗ Let
of the function L(β).
L(β0 , β1 , β2 ) =
n
X
(yi − β0 − β1 xi1 − β2 xi2 )2 .
i=1
5
Solve
∂L
= 0, i = 0, 1, 2. We obtain:
∂βi
n
X
(yi − β0 − β1 xi1 − β2 xi2 ) = 0
i=1
n
X
xi1 (yi − β0 − β1 xi1 − β2 xi2 ) = 0
i=1
n
X
xi2 (yi − β0 − β1 xi1 − β2 xi2 ) = 0
i=1
These equations are also called normal equations. They can be rewritten as:
n
X
yi = β0 n + β1
i=1
n
X
i=1
n
X
n
X
xi1
+ β2
n
X
i=1
xi1 yi = β0
xi2 yi = β0
i=1
n
X
i=1
n
X
xi2
i=1
n
n
X
X
i 2
i
xi1 xi2
(x1 ) + β2
x1 + β 1
xi2 + β1
i=1
i=1
n
X
xi1 xi2 + β2
i=1
(1)
i=1
n
X
(xi2 )2
i=1
The coefficients β0 , β1 , β2 can be computed by solving systems of linear Equations (1).
Now suppose another set of k data points are to be added:
(xn+1
, xn+1
, yn+1 ), . . . , (xn+k
, x2n+k , yn+k ).
1
2
1
The Equations (1) becomes
n+k
X
xi2
i=1
n+k
X
xi1 xi2
i=1
n+k
X
xi2 xi2
i=1
= β0 (n + k) + β1
= β0
= β0
n+k
X
i=1
n+k
X
xi1 + β1
xi2 + β1
i=1
n+k
X
xi1
+ β2
n+k
X
xi2
i=1
n+k
X
i=1
n+k
X
i=1
n+k
X
i=1
n+k
X
(xi1 )2 + β2
xi1 xi2 + β2
i=1
xi1 xi2
(xi2 )2
i=1
The new sum can be expressed as the old sum plus extra terms.
n+k
X
xi2 =
i=1
n+k
X
xi1 xi2
n
X
i=1
=
n
X
i=1
i=1
n+k
X
n
X
(xi1 )2
yi +
=
i=1
n+k
X
xi2 ,
i=n+1
xi1 xi2
+
n+k
X
xi1 xi2 ,
i=n+1
(xi1 )2
+
n+k
X
(xi1 )2 .
i=n+1
i=1
and so on, for the other summations.
6
To generalize the method described above, it is more convenient to rewrite these
equations by using matrices.
   

β0
1 x11 x12 . . . x1m
y1
1 x2 x2 . . . x2   β1   y2 
m 
2
1

  
 .. ..
.. . .
..   ..  =  ..  .
. .
. .  .   . 
.
1 xn1 xn2 . . . xnm
βm
yn
The relationship between matrix X and vector ⃗y can also be written in matrix form
as
X β⃗ = ⃗y .
(2)
The loss function can be rewritten as:
⃗ = ||X β⃗ − ⃗y ||2
L(β)
T = X β⃗ − ⃗y
X β⃗ − ⃗y
⃗
= ⃗y T ⃗y − ⃗y T X β⃗ − β⃗ T X T ⃗y + β⃗ T X T X β.
The optimum solution is at gradient 0. The gradient of the loss function is:
T
T
T
T
T
T
⃗
⃗
⃗
⃗
∂
⃗
y
⃗
y
−
⃗
y
X
β
−
β
X
⃗
y
+
β
X
X
β
⃗
∂L(β)
=
∂ β⃗
∂ β⃗
⃗
= −2X T ⃗y + 2X T X β.
Setting the gradient to 0, we get the optimum coefficients:
− 2X T ⃗y + 2X T X β⃗ = 0
⇒ X T ⃗y = X T X β⃗
−1 T
⇒ β⃗ = X T X
X ⃗y .
Therefore the optimum value for β⃗ is
β⃗ = X T X
−1
X T ⃗y .
(3)
Suppose that each user u has a set of (xi1 , xi2 , . . . , xim ) and its response yi . They want to
collaboratively compute the optimal coefficients
β⃗ = (β0 , β1 , . . . , βm ) ,
such that ||X β⃗ − ⃗y ||2 is minimized. Assume that there are two parties A and B. A’s
data is in the
parts of X and B’s data is in the lower
upper
part of X. Matrix X can be
X1
⃗y
written as
. Similarly, vector ⃗y can be written as 1 .
X2
⃗y2
The system of Equation 2 can be written as:
X1 ⃗
⃗y
β= 1
X2
⃗y2
7
Apply Equation 3, the value of β⃗ can be computed as
T !−1 T X1
X1
X1
⃗y1
β⃗ =
X2
X2
X2
⃗y2
T T X1 −1 T
X1 ⃗y1 + X2T ⃗y2
= X1 X 2
X2
T
−1 T
X1 ⃗y1 + X2T ⃗y2
= X1 X1 + X2T X2
⃗ the matrix needs to be inverted becomes the
To compute the model parameter β,
(m + 1) × (m + 1) matrix
T
X1 X1 + X2T X2 .
To be used in practical applications, the matrices that are to be maintained are X T X
and X T ⃗y . These matrices are initialized as zero matrices. Whenever a new dataset Xi
and ⃗yi are obtained, the new matrices XiT Xi and XiT ⃗yi are added to X T X and X T ⃗y ,
respectively. This enables the computation of a new set of model parameters for linear regression. It is also possible to delete a particular dataset, such as Xj and ⃗yj , by
subtracting XjT Xj and XjT ⃗yj from X T X and X T ⃗y , respectively.
4.2. The Protocol
Suppose that A and B intend to utilize the least square method to construct a linear
model that comprises of m features. Assume that A has the data X1 and the associated
vector ⃗y1 , B has the data X2 and the associated vector ⃗y2 . Our secure collaborative
computation of the optimal parameter for linear regression is shown in Figure 1.
1. A perform n1 experiments and collects the data into an n1 × (m + 1) matrix X1
and a n1 × 1 vector ⃗y1 .
2. B perform n2 experiments and collects the data into an n2 × (m + 1) matrix X2
and a n2 × 1 vector ⃗y2 .
3. A sends X1T X1 and X1T ⃗y1 to B.
4. B sends X2T X2 and X2T ⃗y2 to A.
⃗ which is an (m +
5. A and B can then compute the optimal model parameter β,
1) × 1 vector:
−1 T
X1 ⃗y1 + X2T ⃗y2 .
β⃗ = X1T X1 + X2T X2
Figure 1: Secure Collaborative learning of linear regression without encryption. In the protocol, assume
that A and B intend to utilize the least square method to construct a linear model that comprises of m
features.
4.3. Analysis of the Protocol
Based on the analysis in Section 4.1, it is easy to verify that our collaborative computation of the optimal parameter for linear regression is correct. For the computational
complexity of the scheme, each party i, i = 1, 2, needs to compute XiT Xi and XiT ⃗yi .
This is necessary even if computations are performed by only one party using all the
data in one set. The only possible extra work is every party needs to invert the matrix
8
⃗ The dimension of the matrix to be inverted is
X1T X1 + X2T X2 to obtain the solution of β.
(m + 1) × (m + 1), where m is the number of features used in the model, which is usually
smaller than number of data n.
This step can also be done by one party (i.e. the server) and then send the solution
⃗
β to the other parties (i.e. local users). In this way, the total computational work is the
same as the computational work required to be done when all the data is in one site.
For communication complexity, our proposed scheme sends only aggregated data to the
other party. The size of the aggregated data is only (m + 1)2 + (m + 1) = (m + 1)(m + 2)
numbers for each party having n records of data. The communication complexity is
proportional to the number of features m, which is independent of n. This is less than
the size of the dataset, which is (n(m + 1) + n), when n ≥ (m + 2).
4.4. Security Analysis
In this section, we show that the security of the proposed scheme is informationtheoretically secure. This implies that even though a minor amount of information is
inevitably divulged to establish a useful linear regression model, the attacker cannot
compute the exact values of the other party’s data, even if the attacker has infinite
computing power.
We assume that all parties are semi-honest, i.e. the participants strictly follow the
protocol, but they are interested in knowing additional information that they are not
explicitly permitted.
Theorem 4.1. Assume that each party collects more than m + 2 data. The proposed
collaborative learning method for the optimal parameters of a linear regression model is
secure in the following sense:
1. A does not have enough information to compute any elements in X2 and ⃗y2 , and
2. B does not have enough information to compute any elements in X1 and ⃗y1 .
Proof. Assume that A has a set of n1 data and B has a set of n2 data, n1 + n2 = n and
n1 , n2 > m + 2, where m is the number of features in the system. First we show that A
cannot compute the data of B (i.e. X2 and ⃗y2 ) by using the information A collects in the
execution of the protocol. The number of elements in X2 is n2 × (m + 1), but the first
column of X2 is a constant. Thus, there are n2 m variables. The number of elements in
⃗y2 is n2 , and it contains n2 variables. Thus, A has totally n2 (m + 1) unknowns.
On the other hand, the number of elements in X2T X2 is (m + 1)2 , and the number of
elements in X2T ⃗y2 is m + 1. The first element in the first row of X2T ⃗y2 is n2 . Thus, A has
only (m + 1)2 + n2 − 1 quadratic equations. Since n2 > m + 2, A does not have enough
information to compute the values of the elements in X2 and ⃗y2 .
Similarly, we can show that B does not have enough information to compute the values
of the elements in X1 and ⃗y1 .
We show that even if the parties or the server have infinite computational power,
there is not enough information to infer the value of the other party’s data. That is, the
security of the proposed linear regression scheme does not depend on computationally hard
problems, such as large integer factorization and solving large sets of discrete logarithm
problems. These computationally hard problems are solvable if the attacker has sufficient
computing power, for example using a quantum computer.
Although we have proved that the curious party cannot compute the values of the
dataset of the other parties, even if they have infinite computing power, but the curious
9
Table 1: Comparisons of related protocols
TA
HE
YG(SMC)
CF
Security
Cock’s [12]
Y
Y
N
Y
C
Gascón’s [10]
Y
Y
Y
N
C
Kikuchi’s [19]
N
Y
N
Y
C
Mandal’s [13]
Y
Y
N
N
C
Mohassel’s [14]
Y
Y
Y
N
C
Qiu’s [11]
Y
Y
Y
N
C
Our’s
N
N
N
Y
P
1.
2.
3.
4.
TA: needs trusted or semi-trusted authority.
HE: uses homomorphic encryption.
YG(SMC): uses Yao’s garbled Circuit.
CF uses closed form formula to compute optimal parameters. (Note that some schemes do not use closed-form formulations but
instead use conjugate gradient descent, but this is computationally inefficient and vulnerable to message chaining attacks.)
5. Security: C means computationally secure, P means for perfect secure.
party does know some information about the dataset. For example, the number of data
and the sum of the squares of the data, etc. For the information leakage in general, we
note that Dwork and Naor show that if a machine learning model is useful, it must reveal
some information about the data on which it was trained [23, 24]. Our proof indicates
that the information disclosed to the other party through the proposed collaborative linear
regression scheme is minimal. As a result, none of the parties can leverage it to calculate
the exact values of each other’s data, even if the attacker has unlimited computing power.
4.5. Comparsions
In this section, we compare several other related schemes for federated learning for linear regression. These schemes include Cock’s scheme [12], Gascón’s scheme [10], Kikuchi’s
scheme [19], Mandal’s scheme [13], Mohassel’s scheme [14] and Qiu’s scheme [11]. They
are recently proposed and related to the security of federated learning for linear regression.
These schemes has several characteristics that can be compared, such as whether the
scheme uses a trusted server, and the assumption of a trusted server is indeed unrealistic
and unrealistic in some cases. Homomorphic encryption is computationally intensive
and can only achieve computational security. Some schemes use Yao’s garbled circuit.
Although this general secure multi-party computing protocol can realize secure computing
for any functions, the amount of computation is also very large. Some of the scheme do not
used closed form for optimum parameter, instead they sent gradient decent iteratively, but
this may suffer privacy leakage attack. Some schemes do not use closed form to obtain
optimal parameters, but require sending gradient descent repeatedly to obtain better
parameters. Repeatedly sending gradient descent may compromise data confidentiality.
We incorporate these features into the comparison, Finally, we compare the security level
of these schemes. Some schemes can achieve information theoretically secure while others
can only be computationally. The results of the comparison is shown in Table 1.
5. Applications of Collaborative Learning Scheme
The proposed collaborative learning method for linear regression can be viewed as a
peer-to-peer mode of collaborative learning, or decentralized federated learning. In addition
to the peer-to-peer model for collaborative learning, the proposed learning scheme can
also be used for the client-server mode of collaborative learning, or centralized federated
learning. In the client-server mode of collaborative learning, the server may or may not
own any training data. Each party sends its training data to the server in the aggregated
form. The server can then do all the computations for the users, and send the final result,
⃗ to each user.
i.e. the value of β,
10
User’s new training data can also be sent to the server many times. When sending
their training data incrementally, care must be taken to avoid sending a batch of too
few data. For example, suppose that initially a user A has collected a set of n1 training
data. If n1 > m + 2, then the user can send the corresponding matrix G1 = X1T X1 and
⃗h1 = X T ⃗y1 to the server. Assume that more training data was collected by the same
1
user A. If the number of new training data n2 > m + 2, then A can send another pair
of matrices G2 = X2T X2 and ⃗h2 = X2 ⃗y2 to the server. In the case that the number of
training data n2 ≤ m + 2, user A may not want to send G2 = X2T X2 and ⃗h2 = X2T ⃗y2 to
the server for security and privacy reasons. This is because the number of training data
in X2 and ⃗y2 is too small, and the curious server can compute the matrix X2 and ⃗y2 from
G2 and ⃗h2 .
For security, it is required that the number of training data ni must be greater than
m+2 for each user i. This is not a serious limitation, because, in any practical application,
the number of features is usually small, and the amount of collected data should be much
larger than the number of features to built an accurate machine learning model.
In the design of the protocol, we do not make any assumptions on how the data is
distributed among the parties. In particular, each party may collect its own data with
different set of features. In 2019, Yang et al. introduced a comprehensive secure federated learning framework, which includes horizontal federated learning, vertical federated
learning, and federated transfer learning [5]. In our protocol, the original data can be
horizontally or vertically partitioned, as long as the set of features are determined, our
protocol can be applied.
Our proposed collaborate learning method can easily be implemented to compute
optimal model parameters for linear regression incrementally. Suppose that k datasets
are collected over a period of time:
(X1 , ⃗y1 ), (X2 , ⃗y2 ), · · · , (Xk , ⃗yk ).
Define an (m + 1) × (m + 1) matrix
Gi = XiT Xi
and an (m + 1) × 1 vector
⃗hi = X T ⃗yi .
i
The optimal model parameters β⃗ can be computed by the equation
β⃗ = G−1⃗h,
where
G=
k
X
Gi and ⃗h =
i=1
k
X
⃗hi .
i=1
These k sets of data (X1 , ⃗y1 ), (X2 , ⃗y2 ), · · · , (Xk , ⃗yk ), can be the training data of k users
collect at different times. When some dataset (Xi , ⃗yi ) are available, they can be added
into G and ⃗h respectively. Better model parameters β⃗ can then be computed.
It is also possible to delete some specified datasets, as long as the relevant aggregated
data is properly preserved. For example, if we want to delete all datasets associated with
a party, we need to know the aggregated data for that party. Aggregated data of that
party may also be held by that party. Once that party wants to delete his entire dataset,
he can send the aggregated data to the server, which can be used to deduct his data from
the system.
11
6. Conclusions
We propose a simple yet effective secure multi-party computation scheme for linear
regression in a distributed environment. In the design of our scheme, we exploit the
mathematical structure of linear regression, so that any party needs to send only the
aggregated data to the server or to the other party. Therefore, security and privacy
can be well protected without encryption. Security and privacy play important roles in
collaborative learning. When data owners do not need to publish the valuable data they
collect, they are more willing to share their data to build a better model.
The proposed scheme also allows incremental addition and deletion of some datasets
used in the linear regression models. New datasets can be easily integrated into the
system to obtain new and more accurate model parameters without having to compute
from scratch. With proper bookkeeping, user’s datasets can also be removed from the
model without referencing the original data being removed. The deletion of some datasets
is useful if some party leaves the group and ask for the deletion of his dataset.
The security of our proposed collaborated computation scheme for linear regression
does not need encryption. It does not depend on computationally hard problems, either.
These computationally hard problems usually require heavy computation to achieve certain level of security. Furthermore, computationally hard problems may be broken if the
attacker has enough computing power, such as using quantum computers. Due to the lack
of sufficient information, we proved that the curious party cannot obtain exact values of
the other party’s data, even with unlimited computing power.
Different weights can also be incorporated into collaborative learning in our new
method. This property may be useful in “customized” machine learning, in which all
parties have similarities, but each party also has its own special characteristics. Handwritten character recognition for smartphone input processing is a good example of a need
for a personalized model. In the proposed new method for collaborative learning, each
data owner handles his data independently. By using our proposed method, customization
can be easily achieved by giving this user more weight.
References
[1] T. E. Union, General data protection regulation (2016).
URL https://gdpr-info.eu/
[2] H. L. Seal, Studies in the history of probability and statistics. xv the historical development of the gauss linear model, Biometrika 54 (1-2) (1967) 1–24.
doi:10.1093/biomet/54.1-2.1.
URL https://doi.org/10.1093/biomet/54.1-2.1
[3] R. Tibshirani, Regression shrinkage and selection via the lasso, Journal of the Royal
Statistical Society. Series B (Methodological) 58 (1) (1996) 267–288.
URL http://www.jstor.org/stable/2346178
[4] J. Fox, Applied regression analysis, linear models, and related methods, Sage Publications, Inc., 1997.
[5] Q. Yang, Y. Liu, T. Chen, Y. Tong, Federated machine learning: Concept and applications, ACM Transactions on Intelligent Systems and Technology 10 (2) (January
2019). doi:10.1145/3298981.
URL https://doi.org/10.1145/3298981
12
[6] P. Paillier, Public-key cryptosystems based on composite degree residuosity classes,
in: J. Stern (Ed.), Advances in Cryptology — EUROCRYPT ’99, Springer Berlin
Heidelberg, Berlin, Heidelberg, 1999, pp. 223–238.
[7] J. H. Cheon, A. Kim, M. Kim, Y. Song, Homomorphic encryption for arithmetic of
approximate numbers, in: T. Takagi, T. Peyrin (Eds.), Advances in Cryptology –
ASIACRYPT 2017, Springer International Publishing, Cham, 2017, pp. 409–437.
[8] M. Babenko, E. Golimblevskaia, Euclidean division method for the homomorphic scheme ckks, in: 2021 IEEE Conference of Russian Young Researchers in Electrical and Electronic Engineering (ElConRus), 2021, pp. 217–220.
doi:10.1109/ElConRus51938.2021.9396347.
[9] F. Chen, T. Xiang, X. Lei, J. Chen, Highly efficient linear regression outsourcing to a cloud, IEEE Transactions on Cloud Computing 2 (4) (2014) 499–508.
doi:10.1109/TCC.2014.2378757.
[10] A. Gascon, P. Schoppmann, B. Balle, M. Raykova, J. Doerner, S. Zahur,
D. Evans, Privacy-preserving distributed linear regression on high-dimensional
data, Proceedings on Privacy Enhancing Technologies 2017 (4) (2017) 345–364.
doi:doi:10.1515/popets-2017-0053.
URL https://doi.org/10.1515/popets-2017-0053
[11] G. Qiu, X. Gui, Y. Zhao, Privacy-preserving linear regression on distributed data by
homomorphic encryption and data masking, IEEE Access 8 (2020) 107601–107613.
doi:10.1109/ACCESS.2020.3000764.
[12] M. d. Cock, R. Dowsley, A. C. Nascimento, S. C. Newman, Fast, privacy preserving
linear regression over distributed datasets based on pre-distributed data, in: Proceedings of the 8th ACM Workshop on Artificial Intelligence and Security, AISec
’15, Association for Computing Machinery, New York, NY, USA, 2015, pp. 3–14.
doi:10.1145/2808769.2808774.
URL https://doi.org/10.1145/2808769.2808774
[13] K. Mandal, G. Gong, Privfl: Practical privacy-preserving federated regressions
on high-dimensional data over mobile networks, in: Proceedings of the 2019
ACM SIGSAC Conference on Cloud Computing Security Workshop, CCSW’19,
Association for Computing Machinery, New York, NY, USA, 2019, pp. 57–68.
doi:10.1145/3338466.3358926.
URL https://doi.org/10.1145/3338466.3358926
[14] P. Mohassel, Y. Zhang, Secureml: A system for scalable privacy-preserving machine
learning, in: 2017 IEEE Symposium on Security and Privacy (SP), 2017, pp. 19–38.
doi:10.1109/SP.2017.12.
[15] A. C. Yao, Protocols for secure computations, in: Twenty-third Annual Symposium
on Foundations of Computer Science, 1982, pp. 160–164. doi:10.1109/SFCS.1982.38.
[16] C. Zhao, S. Zhao, M. Zhao, Z. Chen, C.-Z. Gao, H. Li, Y. an Tan, Secure multi-party
computation: Theory, practice and applications, Information Sciences 476 (2019)
357–372. doi:https://doi.org/10.1016/j.ins.2018.10.024.
URL https://www.sciencedirect.com/science/article/pii/S0020025518308338
13
[17] Y. Ishai, E. Kushilevitz, A. Paskin, Secure multiparty computation with minimal
interaction, in: T. Rabin (Ed.), Advances in Cryptology – CRYPTO 2010, Springer
Berlin Heidelberg, Berlin, Heidelberg, 2010, pp. 577–594.
[18] R. Ma, Y. Li, C. Li, F. Wan, H. Hu, W. Xu, J. Zeng, Secure multiparty computation for privacy-preserving drug discovery, Bioinformatics 36 (9) (2020) 2872–2880.
doi:10.1093/bioinformatics/btaa038.
URL https://doi.org/10.1093/bioinformatics/btaa038
[19] H. Kikuchi, C. Hamanaga, H. Yasunaga, H. Matsui, H. Hashimoto, C.I. Fan, Privacy-preserving multiple linear regression of vertically partitioned
real medical datasets, Journal of Information Processing 26 (2018) 638–647.
doi:10.2197/ipsjjip.26.638.
[20] L. Zhu, Z. Liu, S. Han, Deep leakage from gradients, CoRR abs/1906.08935 (2019).
arXiv:1906.08935.
URL http://arxiv.org/abs/1906.08935
[21] G. Xu, H. Li, S. Liu, K. Yang, X. Lin, Verifynet: Secure and verifiable federated
learning, IEEE Transactions on Information Forensics and Security 15 (2020) 911–
926. doi:10.1109/TIFS.2019.2929409.
[22] J. B. Ramsey, Tests for specification errors in classical linear least-squares regression
analysis, Journal of the Royal Statistical Society: Series B (Methodological) 31 (2)
(1969) 350–371. doi:https://doi.org/10.1111/j.2517-6161.1969.tb00796.x.
[23] C. Dwork, M. Naor, On the difficulties of disclosure prevention in statistical databases
or the case for differential privacy, Journal of Privacy and Confidentiality 2 (1)
(September 2010). doi:10.29012/jpc.v2i1.585.
[24] C. Dwork, A. Roth, The algorithmic foundations of differential privacy, Foundations and Trends Theoretical Computer Science 9 (3–4) (2014) 211–407.
doi:10.1561/0400000042.
14