Abstract
We develop and analyze a covariate shift adaptation method based on pseudo-labeling. The goal is to learn a regression function with small mean squared error over a target distribution, based on unlabeled data from there and labeled data that may have a different feature distribution. We propose to split the labeled data into two subsets and run regression on them separately to obtain (1) a collection of candidate models induced by different hyperparameters, and (2) an imputation model. We use the latter to fill the missing labels and then select the best candidate model accordingly. To investigate the influence of pseudo-labels on model selection, we derive a bias-variance decomposition that highlights the importance of using an imputation model with low bias. We demonstrate the efficacy of our approach through kernel ridge regression, proving that our method effectively adapts to the unknown covariate shift.