File size: 76,026 Bytes

f71c233

# KL GUIDED DOMAIN ADAPTATION

**A. Tuan Nguyen[∗†]** **Toan Tran** _[‡]_ **Yarin Gal** _[†]_

**Philip H. S. Torr** _[†]_ **Atılım G¨unes¸ Baydin** _[†]_

ABSTRACT

Domain adaptation is an important problem and often needed for real-world applications. In this problem, instead of i.i.d. training and testing datapoints, we
assume that the source (training) data and the target (testing) data have different
distributions. With that setting, the empirical risk minimization training procedure often does not perform well, since it does not account for the change in
the distribution. A common approach in the domain adaptation literature is to
learn a representation of the input that has the same (marginal) distribution over
the source and the target domain. However, these approaches often require additional networks and/or optimizing an adversarial (minimax) objective, which
can be very expensive or unstable in practice. To improve upon these marginal
alignment techniques, in this paper, we first derive a generalization bound for the
target loss based on the training loss and the reverse Kullback-Leibler (KL) divergence between the source and the target representation distributions. Based on this
bound, we derive an algorithm that minimizes the KL term to obtain a better generalization to the target domain. We show that with a probabilistic representation
network, the KL term can be estimated efficiently via minibatch samples without
any additional network or a minimax objective. This leads to a theoretically sound
alignment method which is also very efficient and stable in practice. Experimental
results also suggest that our method outperforms other representation-alignment
approaches.

1 INTRODUCTION

With advances in neural network architectures (He et al., 2016; Vaswani et al., 2017), machine
learning algorithms have achieved state-of-the-art performance in many tasks such as object classification, object detection and natural language processing. However, machine learning models have
been focusing mostly on the case of independent and identically distributed (i.i.d.) datapoints; and
such an assumption often does not hold in practice. When the i.i.d. assumption is violated and the
target domain has a different distribution compared to the source domain, a typical learner trained
on the source data via empirical risk minimization would not perform well at test time, since it does
not account for the distribution shift. To tackle this problem, many methods have been proposed for
domain adaptation (Zhao et al., 2019; Zhang et al., 2019; Combes et al., 2020; Tanwani, 2020) and
domain generalization (Khosla et al., 2012; Muandet et al., 2013; Ghifary et al., 2015), the goal of
which is to train a machine learning algorithm that can generalize well to the target domain.

A common approach to tackle these problems is to learn a representation such that its distribution
does not change across domains. There are two types of distribution alignment: marginal alignment
(aligning the marginal distribution of the representation) and conditional alignment (aligning the
conditional distribution of the label given the representation) (Nguyen et al., 2021; Tanwani, 2020).
For domain adaptation and domain generalization problems with multiple source domains, we can
use the data and labels to align both the marginal and the conditional distributions across the source
domains, aiming to generalize to the target domain. However, in a single-source domain adaptation
problem, with only unlabeled data from the target domain, it is often only possible to align the
marginal distribution of the representation. This marginal alignment should help the classifier avoid
out-of-distribution data at test time.

_∗Corresponding author: A. Tuan Nguyen, tuan@robots.ox.ac.uk_
_†University of Oxford, Oxford, United Kingdom_
_‡VinAI Research, Hanoi, Vietnam_


-----

This paper focuses on such a single-source domain adaption problem, which is also one of the
most common settings in practice. Current marginal alignment techniques usually require additional computation (e.g., of an additional network) (Ganin et al., 2016; Li et al., 2018) and/or a
minimax objective (Ganin et al., 2016; Shen et al., 2018), leading to an expensive and/or unstable
training procedure (Goodfellow, 2016; Kodali et al., 2017). For example, DANN (Ganin et al., 2016)
employs an adversarial training procedure, with a domain discriminator that classifies the domain
of the representation, and maximizes the adversarial loss of the discriminator. When the discriminator is completely fooled, the marginal distribution of the representation is aligned across domains.
MMD (Gretton et al., 2012; Li et al., 2018) utilizes maximum mean discrepancy to align the representation distribution. This does not use a minimax objective, thus leading to a more stable training;
however, it does require additional computation of several Gaussian kernels. While more sophisticated (non-marginal-alignment) methods have been proposed recently to achieve better results in
the domain adaptation problem, we argue that studying the family of plain marginal-alignment techniques is still an important task, since they are the backbones that most other domain adaptation (and
domain generalization) methods are built upon.

To address the above issues of existing marginal-alignment techniques, we first derive a generalization bound on the loss of the target domain using the training loss and a reverse Kullback–Leibler
(KL) divergence between the source and target distributions. There are existing bounds of the target
loss in the literature (Ben-David et al., 2010), however, these analyses focus mostly on the case of
binary classification and the bounds use a total variation distance or a H-divergence between the
distributions, which are not easy to estimate in practice (for example, Ajakan et al. (2014) require
an adversarial network to estimate the H-divergence). In this paper, we show that with a probabilistic representation network, we can estimate the KL divergence easily using samples, leading
to an alignment method that requires virtually no additional computation nor a minimax objective.
Therefore, our training procedure is simple and stable in practice. Moreover, the reverse KL has the
zero-forcing effect (Minka et al., 2005), which is very effective to alleviate the out-of-distribution
problem in practice. This can be explained as follows: the out-of-distribution problem arises when
the classifier faces a new representation at test time that is in a (near) zero mass region of the source
representation distribution (and thus it never faced before). The reverse KL tends to force the target
representation distribution to have (near) zero mass wherever the source distribution has (near) zero
mass (this is the zero-forcing property), which helps the classifier avoid out-of-distribution data. The
reverse KL also has the mode-seeking effect (Minka et al., 2005) which allows for a more flexible
alignment of the representation (to one or some of the modes of the source domain). For example,
consider the classification problem of buildings (houses, hotels, etc.) where source images are collected from urban and remote areas of a country (two modes); while the target images are collected
from urban areas but from a different country. Ideally, we want to match the representation distribution of the target domain to that of the first mode of the source domain since they are both from
urban areas. The reverse KL allows this flexible alignment (as it results in a relatively small value
of the reverse KL) due to its mode-seeking property. Meanwhile, other distance metrics/divergences
aim to match the whole source and target representation distribution, which might collapse the two
modes of the source domains.

Our contributions in this work are:

-  We construct a generalization bound of the test loss in the domain adaptation problem using
the reverse KL divergence.

-  We propose to reduce the generalization bound by minimizing the above KL term. Furthermore, we show that with a probabilistic representation, the KL term can be estimated easily
using minibatches, without any additional computation or a minimax objective as opposed
to most existing works.

-  We conduct extensive experiments and show that our method significantly outperforms
relevant baselines, namely ERM (Bousquet et al., 2003), DANN (Ganin et al., 2016), MMD
(Gretton et al., 2012; Li et al., 2018), CORAL (Sun and Saenko, 2016) and WD (Shen
et al., 2018). We empirically show that the reverse KL divergence is very effective for
representation alignment since it is very stable and efficient to compute in practice.

2 RELATED WORK

**Generalization bound for the distribution shift problem** There exist works studying bounds for
the distribution shift problem in the literature (Ben-David et al., 2010; Mansour et al., 2009). How

-----

ever, their analyses of the classification problem are limited to the case of binary labels. Moreover,
these bounds are only applicable or practical for deterministic labeling functions, which is not the
case for most datasets in practice. Therefore, their analyses cannot be generalized to the general case
of supervised learning. The differences between our bound and theirs are as follows. First of all, our
bound works for the general case of supervised learning: it works for both the classification (including multiclass classification) and regression problems, it makes no assumptions about the labeling
mechanism (can be probabilistic or deterministic), and it works for virtually all predictive distributions commonly used in practice. Secondly, our bound uses a different divergence, namely the KL
divergence, which is easier to estimate in practice compared to total variation or H-divergence. We
provide a brief review of the above bounds and discuss their differences to ours in more detail in the
appendix. Recently, Acuna et al. (2021) revise the previous domain adaptation bounds and generalize them to a multi-class classification setting, as well as to the class of f -divergence (including
KL divergence). However, their setting is still very restricted: the loss function needs to satisfy the
triangle inequality (which does not hold for many loss functions in practice), and they need to know
the true labeling function (optimal Bayes classifier) for a probabilistic labeling mechanism (which is
often not available). We also provide further analysis, that under some reasonable assumptions, the
conditional misalignment in the representation space is bounded by the conditional misalignment
in the input space, which allows for a sound marginal alignment method. This can be seen as an
improvement over prior works. Some specific cases of distribution shift have also been studied.
For example, Cortes et al. (2010) and Johansson et al. (2019) study the generalization bound for
the covariate shift problem, i.e., pT (x) ̸= pS(x) but pT (y|x) = pS(y|x), where pS is the source
distribution and pT is the target distribution. In contrast, Azizzadenesheli et al. (2019) provide a
generalization bound for the label shift problem, i.e., pT (y) ̸= pS(y) but pT (x|y) = pS(x|y).

**Domain adaptation** While the literature on the domain adaptation problem is vast, we cover the
most closely related works to ours here. A common method for the domain adaptation problem
is to align the marginal distribution of the representation between the source and target domains.
DANN (Ganin et al., 2016) employs a domain discriminator to classify the domain of a representation and maximizes its adversarial loss (a minimax game). WD (Shen et al., 2018) uses a neural
network function f (which is 1-Lipschitz continuous) to calculate the Wasserstein distance between
two distributions and minimizes it. This is also a minimax game since the Wasserstein distance is
the supremum over the search space of f . MMD (Gretton et al., 2012; Li et al., 2018) uses the
maximum mean discrepancy to align the representation distribution. This method does not need a
minimax objective; however, it requires the additional computation of several Gaussian kernels. Finally, CORAL (Sun and Saenko, 2016) matches the first two moments of the distribution; and while
being a simple method, it fails to align more complex distributions. We consider these marginal
alignment techniques our main baselines since our method falls into this category, and investigate
the effectiveness of the reverse KL divergence in aligning the distributions of representation. Recently, more sophisticated alignment methods (Kang et al., 2019; Xu et al., 2019; Zhu et al., 2020)
have been proposed for the domain adaptation problem, which achieve state-of-the-art performance.
Instead of simply aligning the marginal distribution of the representation, these methods minimize
the intra-class distance of the representation across domains, and possibly maximize the inter-class
distance between them, using the MMD or L2 distance. However, they require pseudo labels for
the target domain (often obtained via clustering). Moreover, they are complementary to our method,
as we conjecture that our method can also be used in conjunction with these, leading to the same
algorithms but with the KL distance instead of MMD or L2.

3 APPROACH

3.1 PROBLEM STATEMENT

In this paper, we consider one of the most common domain adaptation settings, which consists of
a single-source domain S with the joint data distribution pS(x, y) and a target domain T with the
data distribution pT (x, y), where x denotes the input sample and y is the label. We assume that
these two domains have the same support sets X _, Y. Regarding the training process of the domain_
adaptation problem, we further denote a labeled dataset of size NS sampled from the source domain
(x[(]S[i][)][, y]S[(][i][)][)]i[N]=1[S] [, where][ (][x][(]S[i][)][, y]S[(][i][)][)][ ∼] _[p][S][(][x, y][)][, and an unlabeled dataset of size][ N][T][ from the target]_
domain (x[(]T[i][)][)]i[N]=1[T] [, where][ x][(]T[i][)] _[∼]_ _[p][T][ (][x][)][.]_

The goal of a typical domain adaptation framework is to train a model with the labeled dataset of
the source domain together with the unlabeled dataset from the target domain, so that the model will


-----

Source Distribution Target Distribution

𝑦 𝑦

𝑥 𝑧 𝑥 𝑧

Joint: 𝑝& 𝑥, 𝑦, 𝑧= 𝑝& 𝑥, 𝑦𝑝(𝑧|𝑥) Joint: 𝑝, 𝑥, 𝑦, 𝑧= 𝑝, 𝑥, 𝑦𝑝(𝑧|𝑥)


Figure 1: Graphical model. Note that the distribution p(z|x) (green edge), corresponding to our representation network, is shared between the source and target domains.

perform decently in the target domain. Note that this is only effective if the labeling mechanism is
not too different between the source and the target domains (Ben-David et al., 2010).

In the domain adaption problem, we expect the changes in the marginal distribution so that
_pS(x) ̸= pT (x), or the conditional distribution so that pS(y|x) ̸= pT (y|x), or both, which often_
render the typical empirical risk minimization training procedure ineffective. This motivates a line
of approaches that learn a representation z of x whose marginal and conditional distributions are
more aligned across the domains and use it for the prediction task, aiming at a better generalization
performance to the target domain.

The general representation learning framework aims to learn a representation z from x with the
mapping p(z|x), which can be deterministic or probabilistic. That latent representation z is expected
to contain the label-related information; and is then used to predict the label y (by a classifier). Note
that since the source and target domains have the same support set for x and share the representation
mapping p(z|x), they also have the same support set for z, denoted by Z. Given the representation z,
we learn a classifier to predict y through the predictive distribution ˆp(y|z) that is an approximation
of the ground truth conditional distribution pS(y _z). During training, the representation network_
_|_
_p(z|x) and the classifier ˆp(y|z) are trained jointly on the source domain and we “hope” that they can_
generalize to the target domain, meaning that both p(z|x) and ˆp(y|z) are kept unchanged between
training and testing. The graphical model of that representation learning process is represented
in Figure 1. In this paper, we consider a probabilistic representation mapping; specifically, the
representation network will output µ(x) and σ[2](x) and p(z|x) = N (z; µ(x), diag(σ[2](x))), where
_N denotes a Gaussian distribution. This can also be thought of as a generalization of a deterministic_
representation, as we recover the deterministic case if σ[2](x) → 0. Also, note that our method is not
limited to the choice of this representation distribution, i.e., the discussion in this section holds for
virtually any other distribution.

The joint distributions of x, y, z for the source and target domains can be represented as follows

_pS(x, y, z) = pS(x, y)p(z|x),_ _pT (x, y, z) = pT (x, y)p(z|x) ._ (1)

and we define the predictive distribution of y given x as

_pˆ(y_ _x) = Ep(z_ _x)[ˆp(y_ _z)] ._ (2)
_|_ _|_ _|_

**Remark 1. On the inference complexity of a probabilistic representation.**
Using a probabilistic representation, we need to sample multiple z from p(z|x) to estimate Eq. 2

during test time. However, this is not a big issue for the representation learning framework, since we
only need to run the representation network p(z|x) (which is usually deep) once to get a distribution
of z. After sampling multiple z from that distribution, we only need to rerun the classifier ˆp(y|z),
which is usually a small network (e.g., often contains one layer). Furthermore, we can also run
_pˆ(y|z) (a small network) in parallel for multiple z to reduce inference time if necessary._

During training, we usually sample a single z from p(z|x) for each x. The training objective is

_ltrain = Ex,y_ _pS_ (x,y),z _p(z_ _x)[_ log ˆp(y _z)]_ (3)
_∼_ _∼_ _|_ _−_ _|_

(this is also the upper bound of EpS (x,y)[− log ˆp(y|x)] via Jensen Inequality)

= EpS (z,y)[− log ˆp(y|z)] (4)

where − log ˆp(y|z) is the loss of a “data point” (z, y). For common choices of the predictive distribution in the classification and regression problems, this is a non-negative quantity. For example,
for a classification problem with a categorical predictive distribution, this becomes the cross-entropy


-----

loss, while for a regression problem with a Gaussian predictive distribution (with a fixed variance),
it becomes the squared error (with an additive constant).

Minimizing ltrain will enforce ˆp(y|z) ≈ _pS(y|z)._

We consider the below two assumptions of the representation z on the source domain:
**Assumption 1. IS(z, y) = IS(x, y), where IS(·, ·) is the mutual information term, calculated on**
_the source domain. In particular:_


_pS(z, y)_
log

_pS(z)pS(y)_


_pS(x, y)_
log

_pS(x)pS(y)_


_IS(z, y) = EpS_ (z,y)


_IS(x, y) = EpS_ (x,y)


(5)


This is often referred to as the “sufficiency assumption” since it indicates that the representation z has
the same information about the label y as the original input x, and is sufficient for this prediction task
(in the source domain). Note that the data processing inequality indicates that IS(z, y) _IS(x, y),_
_≤_
so here we assume that z contains maximum information about y.
**Remark 2. Assumption 1 is an optimization goal of the training process on the source domain.**
In particular, ltrain (with an additive constant) is an upper bound of −IS(z, y), which is an upper
bound of −IS(x, y). Thus, minimizing ltrain will enforce IS(z, y) to be equal to IS(x, y). For a
more detailed discussion of this, please refer to, for example, Alemi et al. (2016).
**Assumption 2. pS(y** _x) = Ep(z_ _x)[pS(y_ _z)]_ _x, y_ _,_
_|_ _|_ _|_ _∀_ _∈X_ _Y_
When this assumption holds, the predictive distribution in Eq. 2 will approximate pS(y _x), as long_
_|_
as ˆp(y _z) approximates pS(y_ _z)._
_|_ _|_

**Remark 3. Assumption 2 is also an optimization goal of the training process on the source domain.**
This is because ltrain is an upper bound of EpS (x,y)[− log ˆp(y|x)], which is an upper bound
of EpS (x,y)[− log pS(y|x)]. Thus, minimizing ltrain will enforce ˆp(y|x) to be equal to pS(y|x).
Therefore, pS(y _x)_ _pˆ(y_ _x) = Ep(z_ _x)[ˆp(y_ _z)]_ Ep(z _x)[pS(y_ _z)]._
_|_ _≈_ _|_ _|_ _|_ _≈_ _|_ _|_

These two assumptions ensure that our network has good performance on the source domain. Note
also that we only make the above two assumptions about the source domain, where we can enforce
them through the training process. We do not make these assumptions about the target domain, since
we have no access to the full target distribution. These two assumptions will also be used to prove
our later theoretical result (Proposition 2).

3.2 KL GUIDED DOMAIN ADAPTATION

Now we will consider the test loss in the domain adaptation problem, and how we can reduce it. The
test loss (of the target domain) is:

_ltest = EpT (x,y)[−_ log ˆp(y|x)] = EpT (x,y)[− log Ep(z|x)[ˆp(y|z)]] (6)

_≤_ EpT (x,y)[Ep(z|x)[− log ˆp(y|z)]] (Jensen Inequality) (7)

= EpT (z,y)[ log ˆp(y _z)]_ (8)
_−_ _|_


Note that if the representation z is invariant (both marginally and conditionally), then pT (z, y) =
_pS(z, y) and Eq. 8 becomes ltrain, and we have a perfect generalization between the source domain_
and the target domain. However, there is no way to guarantee the invariance, since we do not
know the target domain and the target data distribution. In that case, we introduce the following
proposition that ensures a generalization bound of the test loss based on the training loss and the KL
divergence:
**Proposition 1. If the loss −** log ˆp(y|z) is bounded by M [1] _∀z ∈Z, y ∈Y, we have:_

_ltest ≤_ _ltrain +_ _√[M]2_ KL[pT (y, z)|pS(y, z)] (9)

p

= ltrain + _[M]_ KL[pT (z) _pS(z)] + EpT (z) [KL[pT (y_ _z)_ _pS(y_ _z)]]_ (10)
_√2_ _|_ _|_ _|_ _|_

1In the classification problem, we can enforce this quite easily by augmenting the output softmax of theq
classifier so that each class probability is always at least exp (−M ). For example, if we choose M = 3 ⇒
exp (for the loss of a datapoint, while remaining the output prediction class.K + 0−.05M, ..., p) ≈ 0C.05 · K, and if the output softmax is + 0.05), where K = 1 − 0. (05p ·1 C, p and2, ..., p C is the number of classes. This ensures the boundC ), we can augment it into (p1 · K + 0.05, p2 ·


-----

_Proof. provided in the appendix._

This bound is similar to other bounds in the literature (e.g., Ben-David et al. (2010)) in the sense
that it also contains the training loss, a marginal misalignment term and a conditional misalignment
term (KL[pT (z) _pS(z)] and EpT (z) [KL[pT (z)_ _pS(z)]] respectively in our case). However, Ben-_
_|_ _|_
David et al. (2010) consider a binary classification problem and their bounds are only practical for a
deterministic labeling function (requires knowing the true labeling function, which is unknown for
a probabilistic labeling mechanism, to compute the bound); while our bound works for the general
case of supervised learning with any labeling mechanism. For a brief review of these bounds and a
detailed discussion about their differences to ours, please refer to the appendix. Note that the bound
in Proposition 1 is also true when applying to the input space directly (e.g., replacing z with x).
However, we are more interested in the bound in the representation space, since we can reduce it by
regularizing the KL term.

To reduce the generalization gap, we want pT (z, y) to be close to pS(z, y). Aligning the marginal
distribution (i.e., pS(z) ≈ _pT (z)) helps the classifier network ˆp(y|z) avoid out-of-distribution data_
since the target representations it faces at test time belong to the source representation distribution
which it was trained on; while aligning the conditional distribution (pS(y|z) ≈ _pT (y|z)) makes sure_
the classifier gives more accurate predictions on the target domain since ˆp(y|z) was trained to approximate pS(y _z). In the domain adaptation problem, since we only have the unlabeled data from_
_|_
the target domain, we often align the marginal distribution of z only. However, one problem is that
the conditional misalignment also depends on the representation z, and when learning a representation z that aligns the marginal, we might accidentally increase EpT (z) [KL[pT (y _z)_ _pS(y_ _z)]] at the_
_|_ _|_ _|_
same time, leading to a net increase in the above generalization bound. For example, what if (and
is it possible that) the conditional misalignment increases to infinity while we learn a representation
_z?_

Therefore, it is crucial that we can bound the above conditional misalignment. The below proposition handles this problem.

**Proposition 2. If Assumption 1 and 2 hold, and if** _[p]p[T]S[ (](x,y[x,y])[)]_ _[<][ ∞]_ _[(i.e., there exists N, which can be]_

_arbitrarily large, such that_ _[p]p[T]S[ (](x,y[x,y])[)]_ _[< N][ ∀][x][ ∈X]_ _[, y][ ∈Y][), we have:]_

EpT (z) [KL[pT (y _z)_ _pS(y_ _z)]]_ EpT (x) [KL[pT (y _x)_ _pS(y_ _x)]]_ (11)
_|_ _|_ _|_ _≤_ _|_ _|_ _|_

_Proof. provided in the appendix._

This shows that the conditional misalignment in the representation space is bounded by the conditional misalignment in the input space. This can also be viewed as an improvement over the analyses
of Ben-David et al. (2010), where it is not clear if the conditional misalignment in the representation
space is bounded or not. It then follows that:


_ltest ≤_ _ltrain +_ _√[M]2_


KL[pT (z) _pS(z)] + EpT (x) [KL[pT (y_ _x)_ _pS(y_ _x)]]._ (12)
_|_ _|_ _|_ _|_


As mentioned earlier, in order for domain adaptation to be effective, we should expect that the labeling mechanism does not change too much (Ben-David et al., 2010). Thus, the conditional misalignment EpT (x) [KL[pT (y _x)_ _pS(y_ _x)]] is often small (and fixed – not dependent on the representation_
_|_ _|_ _|_
_z). Therefore, to reduce the generalization bound, we can focus on minimizing KL[pT (z)|pS(z)],_
with the objective:
_ltrain + βKL[pT (z)|pS(z)]_ (13)
where β is a hyper-parameter.

**Discussion on the use of reverse KL:** Our derivation leads to the reverse KL term
KL[pT (z)|pS(z)] as a regularizer of the distance between the two domains representations. We
argue that there are several reasons that make this a good choice as a divergence between the source
and target representation distributions. (1) First of all, as mentioned earlier, the KL term can be computed easily without any additional network or a minimax objective (details in Subsection 3.3). This
leads to an efficient and stable training procedure, which often results in improved performance. (2)
Secondly, the reverse KL has the zero-forcing/mode-seeking effect (Minka et al., 2005) that helps to
alleviate the out-of-distribution problem. Specifically, the reverse KL forces the target representation
distribution to have zero mass wherever the source distribution has zero mass (zero-forcing), thus


-----

(a) (b) (c)


Figure 2: Reverse KL allows a flexible alignment of the representation while still effectively preventing the
out-of-distribution problem. (a) Source representation distribution (black). Consider the case where the data
distribution pS(x) of the source domain has two modes, then the representation distribution pS(z) will likely
also have two modes; and consider the case where the target distribution has only one mode. (b) An acceptable
target representation distribution (green) that helps the classifier avoid the out-of-distribution problem. Reverse
KL allows for this type of flexible alignment (match to one/some of the modes) due to its mode-seeking nature.
**(c) A problematic target representation distribution (red), since the classification network will face out-of-**
distribution data at test time, in the area between the two modes. Reverse KL will prevent this due to its
zero-forcing nature.

preventing the out-of-distribution data at test time (Figure 2c). On the other hand, its mode-seeking
nature allows flexible alignment of the representation. For example, consider the case where the
source domain is a mixture of two components (Figure 2a, i.e., it has two modes), and the target distribution is close to one of the two components. Ideally, we want to learn a representation network
that matches the representation of the target domain to that of the corresponding component on the
source mixture (Figure 2b). This representation will still perform well at test time since we would
not have the out-of-distribution problem (the classification network is already trained on this mode
of the source distribution). This flexible alignment (to one or some of the modes) is accepted by the
reverse KL since it leads to a relatively small reverse KL value. Meanwhile, other methods such as
DANN, MMD, CORAL and WD aim to match the representation distribution of the target domain
and that of the whole source domain together, which could compress the representation too much,
negatively affecting its expressive power. For instance, in the above example, trying to match the
whole distribution of source and target domains based on other distance metrics might force the two
modes of the source domain to collapse. The flexible alignment of the reverse KL (while still being
very effective to prevent out-of-distribution data) might be beneficial in some practical cases.

We also empirically found that adding an auxiliary term KL[pS(z)|pT (z)] (forward KL) with a small
coefficient βaux to the objective can help to align the distribution faster, leading to the objective:

_ltrain + βKL[pT (z)|pS(z)] + βauxKL[pS(z)|pT (z)]_ (14)

In practice, setting βaux to a (very) small value or zero often leads to the best results (using larger
_βaux can hurt the performance). Note that this does not invalidate our earlier discussion, as with_
such small values of βaux, the alignment behavior is still dominated by the reverse KL.

3.3 OPTIMIZATION

In practice, we estimate Eq. 14 using minibatches. In particular, given a labelled minibatch
(˜x[(]S[i][)][,][ ˜]yS[(][i][)][)]i[B]=1 [of the source domain and an unlabelled one][ (˜]x[(]T[i][)][)]i[B]=1 [of the target domain, and a]
single sampled representation for each x: (˜zS[(][i][)][)]i[B]=1 [and][ (˜]zT[(][i][)][)]i[B]=1[, we can get an unbiased estimator]
of the objective 14 as follows:

_ltrain + βKL[pT (z)|pS(z)] + βauxKL[pS(z)|pT (z)]_
=EpS (z,y)[ log ˆp(y _z)] + βEpT (z)[log pT (z)_ log pS(z)] + βauxEpS (z)[log pS(z) log pT (z)]
_−_ _|_ _−_ _−_


_≈_ _B[1]_


_B_

log ˆp(˜yS[(][i][)] _zS[(][i][)][) +][ β][ 1]_
_i=1_ _−_ _[|][˜]_ _B_

X


log pT (˜zT[(][i][)][)][ −] [log][ p][S][(˜]zT[(][i][)][)]


_i=1_


log pS(˜zS[(][i][)][)][ −] [log][ p][T][ (˜]zS[(][i][)][)] (15)
i


+ βaux


_i=1_


-----

However, it still requires knowing pS(z) and pT (z) to compute Eq. 15. We also use the minibatch
to approximate these quantities:


_pS(z) = EpS_ (x)[p(z _x)]_
_|_ _≈_ _B[1]_


_B_

_p(z_ _x[(]S[i][)][);]_ _pT (z) = EpT (x)[p(z_ _x)]_
_|_ _|_ _≈_ _B[1]_
_i=1_

X


_p(z|x[(]T[i][)][)][.][ (16)]_
_i=1_

X


Intuitively, we use a minibatch of data to construct a distribution of the representation z (which is a
mixture of B components), and match that distribution for the two domains with the KL divergence.
As mentioned earlier, we use a Gaussian distribution with a diagonal covariance matrix for p(z|x)
in practice, and employ the reparameterization trick (Kingma and Welling, 2013) to sample z.

Although the estimator in Eq. 15 is unbiased, the approximations in Eq. 16 will introduce some bias
into our estimator (however, the estimator is still consistent, i.e., it becomes exact when B →∞).
Therefore, the batch size might have an effect on the performance of the model. However, via an
ablation study, we found that the effect of this bias estimator is not severe in practice, and our model
achieves good performance even with a batch size of 64. For detailed results and discussion of this
ablation study, please refer to the appendix C.4.

4 EXPERIMENTS

4.1 DATASETS

**RotatedMNIST consists of 70,000 MNIST (LeCun et al., 2010) images that are divided into**
six domains, each with 11,666 images. The images in each domain are rotated counterclockwise by 0[◦], 15[◦], 30[◦], 45[◦], 60[◦] and 75[◦] respectively. We denote the six domains as
_M0, M15, M30, M45, M60 and M75. We use M0 as the source domain, and perform five experi-_
ments, each with M15, M30, M45, M60 or M75 as the target domain. The task is classification of
the ten digit labels.

**DIGITS is a common domain adaptation dataset, with 3 digit classification sub-datasets, namely**
MNIST, USPS (Hull, 1994) and SVHN (Netzer et al., 2011). Three common adaptation experiments
are MNIST → USPS, USPS → MNIST and SVHN → MNIST.

**VisDA17 (Peng et al., 2017) is a challenging real-world classification dataset with a simulation-to-**
real adaptation task. This dataset contains over 280K images from 12 classes. The source domain
contains renderings of 3D models, while the target domain contains real images.

Please refer to the appendix for results of datasets with more domains such as PACS (Li et al.,
**2017).**

4.2 BASELINES

We consider all common marginal alignment methods for domain adaptation as our baselines, including DANN, MMD, CORAL and WD. We also consider ERM (empirical risk minimization)
and its variant ERM (prob) (same as ERM but with the probabilistic representation network used
in our model). For ERM, DANN, MMD and CORAL, we follow the implementation by Gulrajani
and Lopez-Paz (2020); while for ERM (prob) and WD, we use our own implementation in Pytorch
(Paszke et al., 2019). Note that we do not include methods that are not from the marginal-alignment
family since they are out of the scope of this paper. For the full description of these baselines, please
[refer to our appendix and the official code at https://github.com/atuannguyen/KL.](https://github.com/atuannguyen/KL)

4.3 EXPERIMENTAL SETTING

For the RotatedMNIST and DIGITS experiments, we use a simple convolutional neural network
with four 3×3 convolutional layers (followed by an average pooling layer) as the representation
network. For VisDA17, we use a Resnet50 as the representation network. Only the last layer of the
representation network differs for a deterministic representation (ERM, DANN, CORAL, MMD,
WD) and a probabilistic one (ERM (prob) and KL (ours)). For a representation of size dz, the
last layer’s dimension of a deterministic representation network is dz, while that of a probabilistic
network is 2 · dz (dz for µ and dz for σ[2]). Please refer to the appendix for the detailed experimental
setting (including data split, hyper-parameter tuning for each model, evaluation protocol, etc.)


-----

Table 1: Rotated MNIST experiments with M0 as the source domain.

Target Domain

Model _M15_ _M30_ _M45_ _M60_ _M75_ Average

ERM 97.5±0.2 84.1±0.8 53.9±0.7 34.2±0.4 22.3±0.5 58.4
ERM (prob) 96.8±0.3 83.2±1.6 51.3±0.9 31.4±1.1 20.7±0.7 56.7
DANN 97.3±0.4 90.6±1.1 68.7±4.2 30.8±0.6 19.0±0.6 61.3
MMD 97.5±0.1 95.3±0.4 73.6±2.1 44.2±1.8 32.1±2.1 68.6
CORAL 97.1±0.3 82.3±0.3 56.0±2.4 30.8±0.2 27.1±1.7 58.7
WD 96.7±0.3 93.1±1.2 64.1±3.3 41.4±7.6 27.6±2.0 64.6

KL (ours) **97.8±0.1** **97.1±0.2** **93.4±0.8** **75.5±2.4** **68.1±1.8** **86.4**

Table 2: DIGITS and VisDA17 experiments.

DIGITS VisDA17


Model M → U U → M S → M Average S → R

ERM 73.1±4.2 54.8±6.2 65.9±1.4 64.6 39.1±0.5
ERM (prob) 70.3±3.2 59.0±8.3 67.6±1.3 65.6 37.2±2.2
DANN 90.7±0.4 91.2±0.8 71.1±0.5 84.3 57.7±1.3
MMD 91.8±0.3 94.4±0.5 82.8±0.3 89.7 62.8±1.1
CORAL 88.0±1.9 83.3±0.1 69.3±0.6 80.2 39.5±4.5
WD 88.2±0.6 60.2±1.8 68.4±2.5 72.3 38.9±4.8

KL (ours) **98.2±0.2** **97.3±0.5** **92.5±0.9** **96.0** **70.6±0.5**

4.4 RESULTS

**RotatedMNIST and DIGITS: Table 1 and Table 2 show the results for the RotatedMNIST and**
DIGITS experiments. It is clear that in these experiments, aligning the representation between
domains does help improve the generalization performance. Among the baselines (DANN, MMD,
CORAL, WD), MMD performs the best, which we attribute to the fact that it does not use a minimax
objective, leading to more stable optimization. Meanwhile, CORAL performs the worst, since it only
matches the first two moments of the distributions and might fail to align complex distributions.
Our method, KL, largely outperforms the baselines, indicating its effectiveness. Visualization of
the representation space also shows that our method aligns the representation better than existing
methods. This visualization can be found in the appendix.

**VisDA17 (Table 2): In this challenging dataset, many marginal alignment techniques (CORAL,**
WD) fail the adaptation task (achieve similar accuracy as the ERM baselines). MMD and KL (ours)
are again the best performers, confirming that a stable training objective is beneficial in practice.
Our method outperforms all other marginal-alignment approaches significantly, suggesting the effectiveness of the KL divergence in representation alignment.

5 CONCLUSION

In conclusion, in this paper, we derive a generalization bound of the target loss in the domain adaptation problem using the reverse KL divergence. We then show that with a probabilistic representation,
the KL divergence can easily be estimated using Monte Carlo (minibatch) samples, without any additional computation or adversarial objective. By minimizing the KL divergence, we can reduce the
generalization bound and have a better guarantee about the test loss. We also empirically show that
our method outperforms relevant baselines with large margins, which we attribute to its simple and
stable training procedure and the mode-seeking/zero-forcing nature of the reverse KL. We conclude
that KL divergence is very effective as a tool for representation alignment. In general, a limitation
of marginal alignment methods (ours included) is that when the conditional distribution changes
significantly from the source domain to the target domain, aligning the marginal would not help the
target domain’s performance. This is also reflected in our generalization bound. For future work, we
would want to investigate the use of KL divergence in other types of alignment. For example, we can
follow the algorithm in Kang et al. (2019) to minimize the intra-class distance of the representation
across domains and maximize the inter-class distance between them, but using the KL divergence
instead of MMD as the distance between representation distributions. Another direction would be
using KL divergence to align the conditional distribution across domains in a multi-source setting.


-----

**Acknowledgments** This work is supported by the UKRI grant: Turing AI Fellowship
EP/W002981/1 and EPSRC/MURI grant: EP/N019474/1. We would also like to thank the Royal
Academy of Engineering and FiveAI.

REFERENCES

D. Acuna, G. Zhang, M. T. Law, and S. Fidler. f-domain-adversarial learning: Theory and algorithms. arXiv preprint arXiv:2106.11344, 2021.

H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, and M. Marchand. Domain-adversarial neural
networks. arXiv preprint arXiv:1412.4446, 2014.

A. A. Alemi, I. Fischer, J. V. Dillon, and K. Murphy. Deep variational information bottleneck. arXiv
_preprint arXiv:1612.00410, 2016._

K. Azizzadenesheli, A. Liu, F. Yang, and A. Anandkumar. Regularized learning for domain adaptation under label shifts. arXiv preprint arXiv:1903.09734, 2019.

S. Ben-David, J. Blitzer, K. Crammer, A. Kulesza, F. Pereira, and J. W. Vaughan. A theory of
learning from different domains. Machine learning, 79(1):151–175, 2010.

J. Bergstra and Y. Bengio. Random search for hyper-parameter optimization. Journal of machine
_learning research, 13(2), 2012._

O. Bousquet, S. Boucheron, and G. Lugosi. Introduction to statistical learning theory. In Summer
_School on Machine Learning, pages 169–207. Springer, 2003._

R. T. d. Combes, H. Zhao, Y.-X. Wang, and G. Gordon. Domain adaptation with conditional distribution matching and generalized label shift. arXiv preprint arXiv:2003.04475, 2020.

C. Cortes, Y. Mansour, and M. Mohri. Learning bounds for importance weighting. In Nips, volume 10, pages 442–450. Citeseer, 2010.

Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F. Laviolette, M. Marchand, and
V. Lempitsky. Domain-adversarial training of neural networks. The journal of machine learning
_research, 17(1):2096–2030, 2016._

M. Ghifary, W. B. Kleijn, M. Zhang, and D. Balduzzi. Domain generalization for object recognition
with multi-task autoencoders. In Proceedings of the IEEE International Conference on Computer
_Vision, pages 2551–2559, 2015._

I. Goodfellow. Nips 2016 tutorial: Generative adversarial networks. _arXiv preprint_
_arXiv:1701.00160, 2016._

A. Gretton, K. M. Borgwardt, M. J. Rasch, B. Sch¨olkopf, and A. Smola. A kernel two-sample test.
_The Journal of Machine Learning Research, 13(1):723–773, 2012._

I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. _arXiv preprint_
_arXiv:2007.01434, 2020._

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings
_of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016._

J. J. Hull. A database for handwritten text recognition research. IEEE Transactions on Pattern
_Analysis and Machine Intelligence, 16(5):550–554, 1994. doi: 10.1109/34.291440._

F. D. Johansson, D. Sontag, and R. Ranganath. Support and invertibility in domain-invariant representations. In The 22nd International Conference on Artificial Intelligence and Statistics, pages
527–536. PMLR, 2019.

G. Kang, L. Jiang, Y. Yang, and A. G. Hauptmann. Contrastive adaptation network for unsupervised
domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern
_Recognition, pages 4893–4902, 2019._

A. Khosla, T. Zhou, T. Malisiewicz, A. A. Efros, and A. Torralba. Undoing the damage of dataset
bias. In European Conference on Computer Vision, pages 158–171. Springer, 2012.


-----

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. _arXiv preprint_
_arXiv:1412.6980, 2014._

D. P. Kingma and M. Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114,
2013.

N. Kodali, J. Abernethy, J. Hays, and Z. Kira. On convergence and stability of gans. arXiv preprint
_arXiv:1705.07215, 2017._

Y. LeCun, C. Cortes, and C. Burges. Mnist handwritten digit database. ATT Labs [Online]. Avail_able: http://yann.lecun.com/exdb/mnist, 2, 2010._

D. Li, Y. Yang, Y.-Z. Song, and T. Hospedales. Deeper, broader and artier domain generalization.
In International Conference on Computer Vision, 2017.

H. Li, S. J. Pan, S. Wang, and A. C. Kot. Domain generalization with adversarial feature learning. In
_Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5400–_
5409, 2018.

Y. Mansour, M. Mohri, and A. Rostamizadeh. Domain adaptation: Learning bounds and algorithms.
_arXiv preprint arXiv:0902.3430, 2009._

T. Minka et al. Divergence measures and message passing. Technical report, Citeseer, 2005.

K. Muandet, D. Balduzzi, and B. Sch¨olkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning, pages 10–18. PMLR, 2013.

Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images
with unsupervised feature learning. 2011.

A. T. Nguyen, T. Tran, Y. Gal, and A. G. Baydin. Domain invariant representation learning with
domain density transformations. arXiv preprint arXiv:2102.05082, 2021.

A. Paszke, S. Gross, F. Massa, A. Lerer, J. Bradbury, G. Chanan, T. Killeen, Z. Lin, N. Gimelshein,
L. Antiga, A. Desmaison, A. Kopf, E. Yang, Z. DeVito, M. Raison, A. Tejani, S. Chilamkurthy,
B. Steiner, L. Fang, J. Bai, and S. Chintala. Pytorch: An imperative style, high-performance
deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alch´e-Buc, E. Fox, and
R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024–8035.
2019.

X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. Visda: The visual domain
adaptation challenge, 2017.

K. Saenko, B. Kulis, M. Fritz, and T. Darrell. Adapting visual category models to new domains. In
_European conference on computer vision, pages 213–226. Springer, 2010._

J. Shen, Y. Qu, W. Zhang, and Y. Yu. Wasserstein distance guided representation learning for domain
adaptation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.

B. Sun and K. Saenko. Deep coral: Correlation alignment for deep domain adaptation. In European
_conference on computer vision, pages 443–450. Springer, 2016._

A. K. Tanwani. Domain-invariant representation learning for sim-to-real transfer. arXiv preprint
_arXiv:2011.07589, 2020._

L. Van der Maaten and G. Hinton. Visualizing data using t-sne. _Journal of machine learning_
_research, 9(11), 2008._

A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.03762, 2017.

X. Xu, X. Zhou, R. Venkatesan, G. Swaminathan, and O. Majumder. d-sne: Domain adaptation
using stochastic neighborhood embedding. In Proceedings of the IEEE/CVF Conference on Com_puter Vision and Pattern Recognition, pages 2497–2506, 2019._

Y. Zhang, T. Liu, M. Long, and M. Jordan. Bridging theory and algorithm for domain adaptation.
In International Conference on Machine Learning, pages 7404–7413. PMLR, 2019.


-----

H. Zhao, R. T. Des Combes, K. Zhang, and G. Gordon. On learning invariant representations for
domain adaptation. In International Conference on Machine Learning, pages 7523–7532. PMLR,
2019.

Y. Zhu, F. Zhuang, J. Wang, G. Ke, J. Chen, J. Bian, H. Xiong, and Q. He. Deep subdomain
adaptation network for image classification. IEEE transactions on neural networks and learning
_systems, 32(4):1713–1722, 2020._

A PROOFS

For the following proofs, we treat the variables as continuous variables and always use the integral. If one or some of the variables are discrete, it is straight-forward to replace the corresponding
integral(s) with summation sign(s) and the proofs still hold.

A.1 PROPOSITION 1

_Proof. We have:_

_ltest_ EpT (z,y)[ log ˆp(y _z)]_ (17)
_≤_ _−_ _|_

= _−_ log ˆp(y|z)pT (z, y)dzdy (18)
Z

= _−_ log ˆp(y|z)pS(z, y)dzdy + _−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy_ (19)
Z Z

= ltrain + _−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy_ (20)
Z

Let A = {(z, y)|pT (z, y) _−_ _pS(z, y) ≥_ 0} and B = {(z, y)|pT (z, y) _−_ _pS(z, y) < 0}, using the fact_
that − log ˆp(y|z) ≥ 0 ∀z ∈Z, y ∈Y we have:

_−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy_ (21)
Z


_−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy +_


_−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy_

(22)


_−_ log ˆp(y|z) [pT (z, y) − _pS(z, y)] dzdy_ (23)

_−_ log ˆp(y|z) |pT (z, y) − _pS(z, y)| dzdy_ (24)

_|pT (z, y) −_ _pS(z, y)| dzdy_ (25)

Z


_≤_ _M_


(since − log ˆp(y|z) ≤ _M_ ) (26)

where |.| is the absolute value.

Here,

and pS(Az, y[|][p][T])[ (]. _[z, y][)][ −]_ _[p][S][(][z, y][)][|][ dzdy][ is also called the total variation of the two distributions][ p][T][ (][z, y][)]_

R


-----

Note that:
_pT (z, y) −_ _pS(z, y)dzdy = 0_ (27)
Z


_pT (z, y) −_ _pS(z, y)dzdy = 0_ (28)

_pS(z, y) −_ _pT (z, y)dzdy_ (29)

_|pT (z, y) −_ _pS(z, y)| dzdy_ (30)


_pT (z, y) −_ _pS(z, y)dzdy +_
Z

_pT (z, y) −_ _pS(z, y)dzdy =_
Z

_|pT (z, y) −_ _pS(z, y)| dzdy =_


_pT (z, y)_ _pS(z, y)_ _dzdy = [1]_
_|_ _−_ _|_ 2


_|pT (z, y) −_ _pS(z, y)| dzdy_ (31)


Therefore:


_|pT (z, y) −_ _pS(z, y)| dzdy_ (32)

_|pT (z, y) −_ _pS(z, y)| dzdy_ (33)


_ltest ≤_ _ltrain + M_

= ltrain + _[M]_


Using the Pinsker’s inequality, we have:
2
_pT (z, y)_ _pS(z, y)_ _dzdy_ 2 _pT (z, y) log_ _[p][T][ (][z, y][)]_ (34)
_|_ _−_ _|_ _≤_ _pS(z, y)_ _[dzdy]_
Z  Z

Therefore, we finally have:

_ltest_ _ltrain +_ _[M]_ 2 _pT (z, y) log_ _[p][T][ (][z, y][)]_ (35)
_≤_ 2 s _pS(z, y)_ _[dzdy]_

Z

= ltrain + _√[M]2_ KL[pT (z, y)|pS(z, y)] (36)

p

Which concludes our proof.

Also note that the KL divergence between pT (z, y) and pS(z, y) can further be decomposed into the
marginal misalignment and conditional misalignment as follow:
KL[pT (z, y) _pS(z, y)] = EpT (z,y)[log pT (z, y)_ log pS(z, y)] (37)
_|_ _−_

= EpT (z,y)[log pT (z) + log pT (y _z)_ log pS(z) log pS(y _z)]_ (38)
_|_ _−_ _−_ _|_

= EpT (z,y)[log pT (z) log pS(z)] + EpT (z,y)[log pT (y _z)_ log pS(y _z)]_
_−_ _|_ _−_ _|_
(39)
= EpT (z)[log pT (z) log pS(z)]
_−_

+ EpT (z) EpT (y|z)[log pT (y|z) − log pS(y|z)] (40)

= KL[pT (z) _pS(z)] + EpT (z) [KL[pT (y_ _z)_ _pS(y_ _z)]]_ (41)
_|_ _|_ _|_ _|_

A.2 PROPOSITION 2


_Proof. According to Assumption 1, we have:_
_IS(z, y) = IS(x, y)_ (42)
_HS(y)_ _HS(y_ _z) = HS(y)_ _HS(y_ _x)_ (43)
_⇔_ _−_ _|_ _−_ _|_
_HS(y_ _z) = HS(y_ _x)_ (44)
_⇔_ _|_ _|_
_⇔EpS_ (z,y)[log pS(y|z)] = EpS (x,y)[log pS(y|x)] (45)

_⇔EpS_ (x,z,y)[log pS(y|z)] = EpS (x,y)[log pS(y|x)] (46)

EpS (x,y) Ep(z _x)[log pS(y_ _z)]_ = EpS (x,y)[log pS(y _x)]_ (47)
_⇔_ _|_ _|_ _|_

EpS (x,y) log pS(y _x)_ Ep(z _x)[log pS(y_ _z)]_ = 0 (48)
_⇔_ _|_ _−_ _|_ _|_
 


-----

According to Assumption 2, ∀x ∈X _, y ∈Y we have:_

_pS(y_ _x) = Ep(z_ _x)[pS(y_ _z)]_ (49)
_|_ _|_ _|_

log pS(y _x) = log Ep(z_ _x)[pS(y_ _z)]_ (50)
_⇔_ _|_ _|_ _|_

log pS(y _x)_ Ep(z _x)[log pS(y_ _z)]_ (51)
_⇒_ _|_ _≥_ _|_ _|_

Since _[p]p[T]S[ (](x,y[x,y])[)]_ _[<][ ∞][, there exists][ N >][ 0][ such that][ p]p[T]S[ (](x,y[x,y])[)]_ _[≤]_ _[N][ ∀][x][ ∈X]_ _[, y][ ∈Y][. Therefore:]_

EpT (x,y) log pS(y|x) − Ep(z|x)[log pS(y|z)] (52)
  _pT (x, y)_

=EpS (x,y) log pS(y _x)_ Ep(z _x)[log pS(y_ _z)]_ (53)

_|_ _−_ _|_ _|_ _pS(x, y)_

_N.EpS_ (x,y) log pS(y _x)_ Ep(z _x)[log pS(y_ _z)]_  (54)
_≤_ _|_ _−_ _|_ _|_
=0 (55)
 


Therefore:

We have:


EpT (x,y) log pS(y|x) − Ep(z|x)[log pS(y|z)] = 0 (56)

EpT (x,y) [log _pS(y_ _x)] = EpT (x,y,z)[log pS(yz)]_ (57)
_⇔_ _|_ _|_

EpT (x,y) [log pS(y _x)] = EpT (z,y)[log pS(y_ _z)]_ (58)
_⇔_ _|_ _|_


EpT (z) [KL[pT (y _z)_ _pS(y_ _z)]]_ EpT (x) [KL[pT (y _x)_ _pS(y_ _x)]]_ (59)
_|_ _|_ _|_ _≤_ _|_ _|_ _|_

EpT (z,y) [log pT (y _z)_ log pS(y _z)]_ EpT (x,y) [log pT (y _x)_ log pS(y _x)]_ (60)
_⇔_ _|_ _−_ _|_ _≤_ _|_ _−_ _|_

Using Eq 58, we now only need to prove that:


EpT (z,y) [log pT (y _z)]_ EpT (x,y) [log pT (y _x)]_ (61)
_|_ _≤_ _|_

_⇔−_ _HT (y|z) ≤−HT (y|x)_ (62)
_⇔HT (y) −_ _HT (y|z) ≤_ _HT (y) −_ _HT (y|x)_ (63)
_⇔IT (z, y) ≤_ _IT (x, y)_ (64)
(always true based on the Data Processing Inequality) (65)

B REVIEW OF EXISTING GENERALIZATION BOUNDS

There have been several works studying the generalization bounds of the Domain Adaptation problem. We briefly review the most important and common ones here with a discussion about their
differences to our proposed bound.

B.1 BEN-DAVID ET AL. (2010)

Ben-David et al. (2010) consider a binary classification problem. Let x be the input with the support
a distributionset X and y be the binary label with the support set PX[s] [over the input][ x][ and the true labeling function] Y = {0, 1}. Consider a source domain with[ f][ s][ :][ X →{][0][,][ 1][}][; and similarly]
a target domain with a distribution PX[t] [over the input][ x][ and the true labeling function][ f][ t][ :][ X →]
_{0, 1}. Note that the authors claim that this labeling function can be probabilistic; in that case, f :_
_X →_ [0, 1] denoting the probability. However, we argue that this probabilistic setting is impractical
since we would not know that true underlying function in order to calculate/estimate the bounds
in practice). Therefore, we found that the bound is only practical for the case of a deterministic
labeling mechanism.

The error of the classifier h, which is also a deterministc labeling function, on the source domain is:

_ϵ[s](h) = Ex_ _PX[s]_ [[][|][h][(][x][)][ −] _[f][ s][(][x][)][|][]][,]_ (66)
_∼_


-----

and similarly for the target domain:

_ϵ[t](h) = Ex_ _P tX_ [[][|][h][(][x][)][ −] _[f][ t][(][x][)][|][]][.]_ (67)
_∼_

Here |.| is the absolute value, which means the loss of a data point is the L1 distance of the labels.

Consider a hypothesis space H and let a classifier h be any function from that space. The first
theorem in Ben-David et al. (2010) offers a bound of the target loss ϵ[t](h) based on the source loss
_ϵ[s](h), and the total variation between PX[s]_ [and][ P][ t]X [, and the difference between the two labeling]
function f _[s]_ and f _[t]:_

**Theorem 1 (Ben-David et al. (2010))**

_ϵ[t](h)_ _ϵ[s](h) + 2d1(PX[s]_ _[, P]X[ t]_ [) +] min (68)
_≤_ _PX_ _PX[s]_ _[,P][ t]X_ _[}][ E][x][∼][P][X]_ [[][|][f][ s][(][x][)][ −] _[f][ t][(][x][)][|][]]_
_∈{_

where d1(PX[s] _[, P]X[ t]_ [)][ is the total variational distance, i.e.,][ d][1][(][P][ s]X _[, P]X[ t]_ [) := sup] _X_ [[][P]X[ s] [(][A][)][−][P][ t]X [(][A][)]][,]
_A∈_
and X is the sigma-field of X (set of all subsets of X ).

In this theorem, the term 2d1(PX[s] _[, P]X[ t]_ [)] presents the marginal misalignment and
minPX _P sX_ _[,P][ t]X_ _[}][ E][x][∼][P][X]_ [[][|][f][ s][(][x][)][ −] _[f][ t][(][x][)][|][]][ is the conditional misalignment.]_
_∈{_

Ben-David et al. (2010) also propose another bound based on a variant of the H-divergence, which
is presented in the following theorem:

**Theorem 2 (Ben-David et al. (2010))**

_ϵ[t](h)_ _ϵ[s](h) + d_ ∆ (PX[s] _[, P]X[ t]_ [) +][ λ][H] (69)
_≤_ _H_ _H_

where the ∆ -divergence d ∆ (PX[s] _[, P]X[ t]_ [)] := suph1,h2 Prx _PX[s]_ [[][h][1][(][x][)] = _h2(x)]_
_H_ _H_ _H_ _H_ _∈H |_ _∼_ _̸_ _−_
Prx _P tX_ [[][h][1][(][x][)][ ̸][=][ h][2][(][x][)]][|][ replaces the total variation to measure the marginal misalignment of the]
_∼_
two domains. Meanwhile, λ = inf _h_ [ϵ[s](h) + ϵ[t](h)] measures the conditional misalignment
_H_ _∈H_
of the two domains (if the two true labeling functions f _[s]_ and f _[t]_ are the same and belong to the
hypothesis space H, this quantity is zero).

The above bounds can also be applied to the representation space (similar to ours), leading to
the same bounds where the input x is replaced by its representation z. However, as mentioned
in the main text, it is not clear if the conditional misalignment in the representation space (e.g.,
minPZ _P sZ_ _[,P][ t]Z_ _[}][ E][z][∼][P][Z]_ [[][|][g][s][(][z][)][ −] _[g][t][(][z][)][|][]][) is bounded or not.]_
_∈{_

**Difference to our bound** First of all, Ben-David et al. (2010) only consider a binary classification
problem. Moreover, as discussed above, the bounds in Ben-David et al. (2010) are only practical
with deterministic labeling mechanisms for both domains. This assumption is hard to be true for
most datasets since the labeling mechanism is usually probabilistic. This makes it not generalizable
to the general case of supervised learning. Furthermore, the loss function is a L1 distance between
the labeling function, which is also not a common choice in practice, which makes it challenging to
generalize to the multiclass classification set-up (even if there exists a deterministic labeling function
for a multiclass dataset, using the L1 loss for the one-hot encoded labels would be unreasonable; the
common loss function in practice for the multiclass classification problem is the cross-entropy loss).
Finally, the total variation and H-divergence might be hard to estimate in practice since it requires
the computation of a supremum.

B.2 MANSOUR ET AL. (2009)

Mansour et al. (2009) consider a more flexible problem set-up than Ben-David et al. (2010). Specifically, instead of Y = {0, 1}, they consider the cases where Y = {0, 1} (for binary classification)
or Y is a measurable subset of R (for regression). Note that their bound still cannot work for multiclass classfication. They also generalize the L1 loss function to a loss function L : Y × Y → R;
however, this loss function must obey the triangle inequality. Although the L1 distance satisfies this
inequality, it is not generally true for other common loss functions in practice (e.g., cross-entropy).
They still consider deterministic labeling function f _[s]_ and f _[t]_ for the source and target domain.

Similar to Ben-David et al. (2010), with a hypothesis h from the hypothesis space H, the error of
the source and target domain are:


-----

_ϵ[s](h) = Ex_ _PX[s]_ [[][L][(][h][(][x][)][, f][ s][(][x][))]] (70)
_∼_

_ϵ[t](h) = Ex_ _P tX_ [[][L][(][h][(][x][)][, f][ t][(][x][))]] (71)
_∼_

For convenience, denote also the error between two labeling function h and h[′] in the source and
target distribution as:

_ϵ[s](h, h[′]) = Ex_ _PX[s]_ [[][L][(][h][(][x][)][, h][′][(][x][))]] (72)
_∼_

_ϵ[t](h, h[′]) = Ex_ _P tX_ [[][L][(][h][(][x][)][, h][′][(][x][))]] (73)
_∼_

(which means ϵ[s](h) = ϵ[s](h, f _[s]) and ϵ[t](h) = ϵ[t](h, f_ _[t]))._

Also, let h[∗][s] and h[∗][t] be the minimizer of ϵ[s](h) and ϵ[t](h) respectively. In particular:

_h[∗][s]_ = arg min _X_ [[][L][(][h][(][x][)][, f][ s][(][x][))]] (74)
_h∈H_ _[ϵ][s][(][h][) = arg min]h∈H_ [E][x][∼][P][ s]

_h[∗][t]_ = arg minh∈H _[ϵ][t][(][h][) = arg min]h∈H_ [E][x][∼][P][ t]X [[][L][(][h][(][x][)][, f][ t][(][x][))]] (75)

Mansour et al. (2009) introduce a generalization bound as follow:

**Theorem 3 (Mansour et al. (2009)) Assume that the loss function L is symmetric and obeys the**
triangle inequality. Then, for any hypothesis h ∈H, the following holds

_ϵ[t](h) ≤_ _ϵ[t](h[∗][t]) + ϵ[s](h, h[∗][s]) + disc(PX[s]_ _[, P]X[ t]_ [) +][ ϵ][t][(][h][∗][s][, h][∗][t][)] (76)

where disc(PX[s] _[, P]X[ t]_ [) := sup]h,h[′]
_H∆H-divergence._ _∈H_ _[|][ϵ][s][(][h, h][′][)][ −]_ _[ϵ][t][(][h, h][′][)][|][, which is a generalized version of the]_

Here, the first term ϵ[t](h[∗][t]) is the ideal target loss (will be zero if the hypothesis space contains ft),
_H_
the second term ϵ[s](h, h[∗][s]) will be zero if we choose h = h[∗][s] (which is the common practice, e.g.,
train the classifier h on the source domain), the third term disc(PX[s] _[, P]X[ t]_ [)][ measures the marginal mis-]
alignment, and the final term ϵ[t](h[∗][s], h[∗][t]) is somewhat an indicator of the conditional misalignment
(becomes zero if f _[s]_ = f _[t]_ _∈H)._

**Difference to our bound** The above bound is based on the ideal target loss, while our bound is
based on the source loss. In practice, we have (an estimate) of the source loss calculated on the
source domain’s training set; meanwhile, the ideal target loss is unknown. This makes the above
bound less useful in practice compared to ours. Furthermore, the above bound has similar problems
as the ones in Ben-David et al. (2010): it does not work for multiclass classification, it assumes
a deterministic labeling mechanism (which does not hold in practice), it assumes the loss function
obeys the triangle inequality (which generally is not true in practice), and it contains terms that are
not easy to compute in practice (supremum and infimum).

C ADDITIONAL EXPERIMENTAL RESULTS

C.1 VISUALIZATION OF THE ROTATEDMNIST EXPERIMENTS

Figure 3 shows the representation space (for the RotatedMNIST experiment with source M0 and
target 45) of our method compared to the baselines, visualized using t-SNE (Van der Maaten and
_M_
Hinton, 2008). We can clearly see that the color clusters (which correspond to the digit classes) of
our method are much more aligned when compared to other baselines such as MMD, DANN and
ERM. This illustrates the effectiveness of our method in aligning the representation in the domain
adaptation problem.

C.2 PACS

Table 3 presents the results for PACS, which is a challenging real-world dataset for domain adaptation/generalization. In this dataset, our model outperforms the ERM baselines by roughly 9%


-----

|Col1|Col2|Col3|Col4|Col5|Col6|0|
|---|---|---|---|---|---|---|
|||||||1 2|
|||||||3 4 5|
|||||||6 7|
|||||||8 9|
||||||||
||||||||
||||||||


TSNE Visualization TSNE Visualization TSNE Visualization TSNE Visualization

30 01 30 60 60

2

20 3 20 40 40

4

10 5 10 20 20

6

0 78 0 0 0

10 9 10

TSNE dim 2 TSNE dim 2 TSNE dim 2 20 TSNE dim 2 20

20 20

40 40

30 30

60 60

40 40

60 40 TSNE dim 120 0 20 40 60 60 40 TSNE dim 120 0 20 40 60 60 40 20TSNE dim 10 20 40 60 80 60 40 20TSNE dim 10 20 40 60 80


(a) KL

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|Col10|
|---|---|---|---|---|---|---|---|---|---|
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||
|||||||||||


TSNE Visualization TSNE Visualization TSNE Visualization TSNE Visualization

60 60

60 60

40 40

40 40

20 20 20 20

0 0 0 0

20 20 20 20

TSNE dim 2 TSNE dim 2 TSNE dim 2 TSNE dim 2

40 40 40 40

60 60

60 60

80 80

80 60 40 TSNE dim 120 0 20 40 60 80 80 60 40 TSNE dim 120 0 20 40 60 80 80 60 40 TSNE dim 120 0 20 40 60 80 60 40 TSNE dim 120 0 20 40 60


(c) DANN


(b) MMD

60

40

20

0

20

TSNE dim 2

40

60

80

60

|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||


|Col1|Col2|Col3|Col4|Col5|Col6|Col7|Col8|Col9|
|---|---|---|---|---|---|---|---|---|
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||
||||||||||



(d) ERM


TSNE Visualization

40 TSNE dim 120 0 20 40


TSNE Visualization

40 TSNE dim 120 0 20 40


TSNE Visualization

40 TSNE dim 120 0 20 40


TSNE Visualization

40 TSNE dim 120 0 20 40


Figure 3: Visualization using t-SNE of the representation space of our method KL and the baselines
MMD, DANN, ERM. For each method, the left subfigure corresponds to the source domain 0
_M_
and the right one corresponds to the target domain 45. Each color represents a digit class.
_M_

on average, indicating the effectiveness of our representation-alignment technique. Our method is
the best performer (with a large margin) on 8 out of 12 experiments, showing a clear benefit over
other representation alignment techniques. Together with our method, MMD again performs the
best among the representation-alignment baselines (DANN, MMD, CORAL and WD), confirming
that a stable training procedure (with no minimax objectives in MMD and our model) is important
and often leads to better results. It is also worth noting that our model still outperforms MMD despite being less computationally expensive (in this implementation, MMD needs to compute seven
Gaussian kernels for each of three pairs of representation sets in each minibatch).

It is interesting that the ERM baselines perform the best in some experiments (e.g., S → C, S → P).
This result also agrees with the one observed in Gulrajani and Lopez-Paz (2020) that domain generalization/adaptation techniques might have negative effects when applied unsuccessfully. It should
be noted that the S (sketch) domain is undoubtedly the most different compared to others (only black
sketch on a white background while other domains have colors), which might explain the difficulty
when learning to transfer between domains.


C.3 OFFICE-31

We also provide additional results on the Office-31 dataset Saenko et al. (2010). Table 4 presents
the performance of our model and two of the best performing baselines, namely MMD and DANN.
Although our method is not state-of-the-art in this dataset (and understandingly so), it significantly
outperforms the relevant baselines considered in this paper.


C.4 ABLATION STUDY: EFFECT OF BATCH SIXE

In this subsection, we conduct an ablation study to investigate the effect of the batch size on our
model’s performance. Table 5 shows the performance of our method on the RotatedMNIST dataset,
with M0 as the source domain and M45 as the target domain and with various choices of the batch
size. As expected, our model’s performance tends to benefit from a bigger batch size, since it would
alleviate the bias of our objective estimator. We therefore recommend increasing the batchsize
whenever possible. However, our model performs well even for a batch size as small as 64 (which
is considered small in this era of deep learning).


-----

Table 3: PACS experiments.

Model

Experiments ERM ERM (prob) DANN MMD CORAL WD KL (ours)

A → C 66.1±1.3 63.5±0.8 71.0±3.2 **79.5±0.4** 62.7±10.4 76.2±0.9 73.1±3.4
A → P 94.3±0.6 93.5±1.3 94.5±0.5 94.5±1.1 86.3±6.8 92.4±1.3 **95.4±1.2**
A → S 53.6±0.8 60.9±3.5 58.6±12.8 62.1±2.0 46.2±3.5 53.9±2.7 **67.4±1.9**
C → A 69.7±1.1 70.8±2.3 76.4±1.7 79.5±3.0 75.9±0.9 69.0±2.1 **83.3±1.1**
C → P 82.0±0.9 81.5±2.1 78.6±3.4 80.8±2.3 78.3±3.6 72.9±8.6 **83.1±7.4**
C → S 72.2±1.4 70.4±1.5 **76.1±1.0** 74.1±1.3 56.9±11.0 48.7±6.1 68.2±0.5
P → A 65.7±2.3 63.3±1.2 68.0±2.7 67.7±1.8 70.0±1.5 62.6±1.5 **75.5±2.5**
P → C 29.1±1.9 27.2±3.3 50.7±5.0 47.4±0.8 47.5±8.6 56.1±1.4 **67.7±1.2**
P → S 38.0±1.0 35.9±2.3 29.3±9.8 59.7±4.8 15.8±5.3 22.3±15.0 **64.5±2.1**
S → A 41.3±6.5 40.9±3.9 39.2±3.5 40.0±3.3 39.1±4.8 36.1±9.5 **48.2±2.4**
S → C 66.7±1.0 **67.9±1.4** 64.3±2.0 65.7±2.3 59.9±1.5 60.5±2.0 63.5±0.4
S → P **49.3±3.3** 46.0±4.7 44.3±4.0 45.1±0.9 37.4±2.7 38.5±5.6 39.1±3.4

Average 60.6 60.2 62.6 66.3 56.3 57.4 **69.1**

Table 4: Office-31 experiments


Model A→D A→W D→A D→W W→A W→D Average

DANN 79.7±0.4 82.0±0.4 68.2±0.4 96.9±0.2 67.4±0.5 99.1±0.1 82.2
MMD 75.5±0.6 73.4±0.4 60.8±1.0 97.4±0.5 61.5±1.1 99.5±0.2 78.0
KL 85.6±0.6 87.9±0.4 70.1±1.1 99.0±0.2 69.3±0.7 100.0±0.0 85.3

Table 5: Ablation study: Effect of batch size. Rotated MNIST experiments with M0 source and M45 target.

|Batch size|256 128 64 32|
|---|---|


|KL (ours)|93.4±0.8 93.4±1.2 93.3±0.3 89.5±0.9|
|---|---|



Table 6: Ablation study: Effect of auxiliary term. RotatedMNIST.

Model _M15_ _M30_ _M45_ _M60_ _M75_ Average

KL (βaux = 0) 97.8±0.5 96.6±0.4 92.0±0.4 68.8±4.6 62.3±4.2 83.5
KL (reported in the paper) 97.8±0.1 97.1±0.2 93.4±0.8 75.5±2.4 68.1±1.8 86.4

Table 7: Ablation study: Effect of auxiliary term. DIGITS and VisDA17.

DIGITS VisDA17


Model M → U U → M S → M Average S → R

KL (βaux = 0) 98.2±0.2 97.1±0.4 90.0±1.1 95.1 67.8
KL (reported in the paper) 98.2±0.2 97.3±0.52 92.5±0.9 96.0 70.5

C.5 ABLATION STUDY: EFFECT OF THE AUXILIARY FORWARD KL

In this subsection, we investigate the contribution of the auxiliary forward KL term, by considering
a variant of our method without this term (βaux = 0). Table 6 and Table 7 show the results for this
ablation experiment. Clearly, most of the improvement comes from the reverse KL regularizer term.
And even without the auxiliary forward KL term, our results are still significantly higher than the
baselines. Note that the auxiliary term requires virtually no extra computation, so we believe adding
the auxiliary term is reasonable.

D DETAILED EXPERIMENTAL SETTINGS

In each experiment, we split both the source and the target data into two portions: 80% and 20%.
We use 80% of the source domain data and 80% of the target domain data (without the labels) as the
training data. We use the remaining 20% of the source data as the validation set, and the remaining
20% of the target domain data as the test set. Note that we do not use the labeled data from the
target domain during training or validation. This evaluation protocol is recommended by Gulrajani
and Lopez-Paz (2020).


-----

D.1 BASELINES

**ERM Bousquet et al. (2003) is the typical empirical risk minimization training procedure, meaning**
that the model is trained normally in the training data and does not account for the distribution shift
(domain adaptation).

**ERM (prob): since we use a probabilistic network, we also include the probabilistic version of**
ERM. This is similar to ERM but uses a probabilistic representation network (same as ours).

**DANN Ganin et al. (2016) utilizes a discriminator to distinguish the representation from the source**
and target domains. It uses an adversarial loss to enforce that the distributions of the representation
from the source domain and the target domain are the same.

**MMD Li et al. (2018) uses the maximum mean discrepancy (MMD) to align the representation’s**
distributions.

**CORAL Sun and Saenko (2016) aligns the representation distributions of the source and target**
domains by matching their first two moments.

**WD Shen et al. (2018) uses the Wasserstein distance to match the distribution of the representation.**

D.2 REPRESENTATION NETWORK USED IN ROTATEDMNIST AND DIGITS

We use a simple CNN as the representation network in this experiment. This network is exactly the
same as the one used in Gulrajani and Lopez-Paz (2020).

Our code is in PyTorch. The network is constructed by the following layers, where
output dim=256 for a deterministic representation network and output dim=512 for a probabilistic representation network (256 for µ and 256 for σ[2]):

-  Conv2d(in channels=1,out channels=64,kernel size=3,stride=1,padding=1)

-  ReLU()

-  GroupNorm(num groups=8,num channels=64)

-  Conv2d(in channels=64,out channels=128,kernel size=3,stride=2,padding=1)

-  ReLU()

-  GroupNorm(num groups=8,num channels=128)

-  Conv2d(in channels=128,out channels=128,kernel size=3,stride=1,padding=1)

-  ReLU()

-  GroupNorm(num groups=8,num channels=128)

-  Conv2d(in channels=128,out channels=output dim,kernel size=3,stride=1,padding=1)

-  ReLU()

-  GroupNorm(num groups=8,num channels=output dim)

-  AdaptiveAvgPool2d(output size=(1,1))

D.3 HYPER-PARAMETERS TUNING

We train each model for 100 epochs. To avoid hyperparameter bias, we tune the hyperparameters
(learning rate, regularizer coefficients, weight decay, representation dimension and dropout rate) for
each method and dataset independently. Following Gulrajani and Lopez-Paz (2020), we perform a
random search (Bergstra and Bengio, 2012) to tune hyperparameters for the baselines. We use the
Adam optimizer Kingma and Ba (2014) for all the models. We re-run each set of hyperparameters
three times. We train all models on an NVIDIA Quadro RTX 6000 GPU.

The readers can also refer to our source code for the experiment setting.

Below are the hyper-parameters considered by the random search in our experiments for each baseline. {.} means a set of hyper-parameters considered, while [., .] means a range of hyper-parameters
considered.

D.3.1 ROTATEDMNIST AND DIGITS

The representation’s dimension is 128 (details in Section D.2). For even more details about the
below hyper-parameters, please refer to our provided code.


-----

-  ERM: learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0, batch size: [8, 512],
number of layers of the classifier (pˆ(y|z)): 1 or 3.

-  ERM (prob): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0, batch size:

[8, 512], number of layers of the classifier (pˆ(y|z)): 1 or 3.

-  DANN Ganin et al. (2016): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0,
batch size: [8, 512], number of layers of the classifier (pˆ(y|z)): 1 or 3, adversarial loss coefficient:

[10[−][2], 10[2]], weight decay of discriminator: [10[−][6], 10[−][2]], number of discriminator steps per generator steps: {1, 2, 4, 8}, grad penalty coefficient: [10[−][2], 10[1]]

-  MMD Li et al. (2018): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0, batch
size: [8, 512], number of layers of the classifier (pˆ(y|z)): 1 or 3, MMD coefficient: [10[−][3], 10[−][1]]

-  CORAL Sun and Saenko (2016): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate:
0.0, batch size: [8, 512], number of layers of the classifier (pˆ(y|z)): 1 or 3, CORAL loss coefficient:

[10[−][3], 10[−][1]]

-  WD Shen et al. (2018): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0, batch
size: [8, 512], number of layers of the classifier (pˆ(y|z)): 1 or 3, wasserstein distance coefficient:

[10[−][2], 10[2]], weight decay of network f : [10[−][6], 10[−][2]], number of f optimization steps per normal
optimization steps: {1, 2, 4, 8}, grad penalty coefficient: [10[−][2], 10[1]]

-  KL (ours): learning rate: [10[−][4][.][5], 10[−][2][.][5]], weight decay: 0.0, dropout rate: 0.0, batch size: 256,
number of layers of the classifier (pˆ(y|z)): 1, β : 0.3, βaux : 0.1

D.3.2 VISDA17 AND PACS

The representation network for VisDA17 is a Resnet50, while the one for PACS is a Resnet18. For
even more details about the below hyper-parameters, please refer to our provided code.

-  ERM: learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]], dropout rate: {0.0, 0.1, 0.5},
batch size: [8, 45], representation dimension: {16, 128, 256, 512}, number of layers of the classifier
(pˆ(y|z)): 1 or 3.

-  ERM (prob): learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]], dropout rate:
_{0.0, 0.1, 0.5}, batch size: [8, 45], representation dimension: {16, 128, 256, 512}, number of lay-_
ers of the classifier (pˆ(y|z)): 1 or 3.

-  DANN Ganin et al. (2016): learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]], dropout
rate: {0.0, 0.1, 0.5}, batch size: [8, 45], representation dimension: {16, 128, 256, 512}, number of
layers of the classifier (pˆ(y|z)): 1 or 3, adversarial loss coefficient: [10[−][2], 10[2]], weight decay of
discriminator: [10[−][6], 10[−][2]], number of discriminator steps per generator steps: {1, 2, 4, 8}, grad
penalty coefficient: [10[−][2], 10[1]]

-  MMD Li et al. (2018): learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]], dropout rate:
_{0.0, 0.1, 0.5}, batch size: [8, 45], representation dimension: {16, 128, 256, 512}, number of layers_
of the classifier (pˆ(y|z)): 1 or 3, MMD coefficient: [10[−][3], 10[−][1]]

-  CORAL Sun and Saenko (2016): learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]],
dropout rate: {0.0, 0.1, 0.5}, batch size: [8, 45], representation dimension: {16, 128, 256, 512},
number of layers of the classifier (pˆ(y|z)): 1 or 3, CORAL loss coefficient: [10[−][3], 10[−][1]]

-  WD Shen et al. (2018): learning rate: [10[−][5], 10[−][3][.][5]], weight decay: [10[−][6], 10[−][2]], dropout rate:
_{0.0, 0.1, 0.5}, batch size: [8, 45], representation dimension: {16, 128, 256, 512}, number of layers_
of the classifier (pˆ(y|z)): 1 or 3, wasserstein distance coefficient: [10[−][2], 10[2]], weight decay of
network f : [10[−][6], 10[−][2]], number of f optimization steps per normal optimization steps: {1, 2, 4, 8},
grad penalty coefficient: [10[−][2], 10[1]]

-  KL (ours): learning rate: 10[−][4], weight decay: [10[−][6], 10[−][2]], dropout rate: 0.0, batch size: 256,
representation dimension: 16, number of layers of the classifier (pˆ(y|z)): 1, β : {0.1, 0.05, 0.001},
_βaux : {0.1, 0.05, 0.01, 0.0}_


-----