In this talk, I will introduce my recent work on collaborative (reinforcement) learning (CL), in which multiple agents work together to learn an objective function. We are particularly interested in a scenario in which agent communication is very expensive. Our goal is to identify the tradeoffs between the speedup of the collaboration and the communication cost among the agents. We convey the following messages using a basic problem in bandit theory as a vehicle: (1) adaptive CL is more powerful than non-adaptive CL; (2) CL with non-IID data is harder than that with IID data; and (3) problems incomparable in the single-agent learning model can be separated in the CL model.