The health, education, and other service applications for robots that assist through primarily social rather than physical interaction are rapidly growing, and so is the research into such technologies. Socially assistive robotics (SAR) aims to address critical areas and gaps in care by automating supervision, coaching, motivation, and companionship aspects of one-on-one interactions with individuals from various large and growing populations, including stroke survivors, the elderly and individuals with dementia, children with autism spectrum disorders, among many others. In this way, roboticists hope to improve the standard of care for large user groups. Naturally, SAR systems pose several ethical challenges regarding their design, implementation, and deployment. This paper examines the ethical challenges of socially assistive robotics from three points of view (user, caregiver, peer) using core principles from medical ethics (autonomy, beneficence, non-maleficence, justice) to determine how intended and unintended effects of a SAR can impact the delivery of care.