People often take a series of nearly redundant pictures to capture a moment or scene. However, selecting photos to keep or share from a large collection is a painful chore. To address this problem, we seek a relative quality measure within a series of photos taken of the same scene, which can be used for automatic photo triage. Towards this end, we gather a large dataset comprised of photo series distilled from personal photo albums. The dataset contains more than 15,000 photos (almost 6,000 series). By augmenting this dataset with ground truth human preferences among photos within each series, we establish a benchmark for measuring the effectiveness of algorithmic approaches to modeling human preferences. We introduce several new approaches for modeling human preference based on machine learning. Our paper also describes applications for the dataset and predictor, including a smart album viewer, automatic photo enhancement, and providing overviews of video clips.