Detection and characterization of rare tumor cells in patients’ blood is important for the diagnosis and monitoring of cancer. The traditional way via fluorescent images requires a series of tedious experimental procedures and often impacts the viability of cells. Here we present a method for label-free detection. The approach uses the convolutional neural network, a powerful image classification, and a machine learning algorithm to perform label-free classification of cells detected in microscopic images of patient blood samples containing white blood cells and cultured cell lines. It requires minimal data pre-processing and has an easy experimental setup. We use the encapsulation technique to fix and locate the target cells after the decision of the pre-trained ML model. Through our experiments, we show that our method can achieve high accuracy in the identification of rare tumor cells without the need for advanced devices or expert users, thus providing a faster and simpler way of counting and identifying rare tumor cells.