Detection and characterization of rare circulating tumor cells (CTCs) in patients’ blood is important for the diagnosis and monitoring of cancer. The traditional way of counting CTCs via fluorescent images requires a series of tedious experimental procedures and often impacts the viability of cells. Here we present a method for label-free detection of CTCs from patient blood samples, by taking advantage of data analysis of bright field microscopy images. The approach uses the convolutional neural network, a powerful image classification and machine learning algorithm to perform label-free classification of cells detected in microscopic images of patient blood samples containing white blood cells and CTCs. It requires minimal data pre-processing and has an easy experimental setup.