Textile reinforced mortar (TRM) is applied on the concrete surface with the aim of strengthening reinforced concrete structures. The performance of the strengthened structural system is directly related to the bond between the existing concrete substrate and the freshly applied TRM layer. This paper presents the results of an experimental study carried out to investigate the significance of concrete surface preparation, performed prior to strengthening, on the bonding behavior of the TRM system. For this purpose, concrete slabs of size (500 mm × 500 mm × 100 mm) were prepared and strengthened using a 10-mm thick TRM layer. After that, the bond performance of the strengthening layer with the concrete slab was assessed using the pull-off test. Three different levels of surface roughening were considered before strengthening: (i) no roughening (regarded as the reference), (ii) low roughening level, and (iii) high roughening level. Two types of textile materials are used in strengthening systems: carbon and polyparaphenylene benzobisoxazole (PBO). A total number of 72 pull-off tests were performed, of which the results were analyzed to examine the significance of the test variables. Results revealed that as the concrete surface is more roughened before strengthening, the bond between concrete substrate and TRM layer becomes stronger. Moreover, the PBO-TRM systems exhibit more desirable bonding behavior compared to the carbon-TRM counterpart.