Personalized recommender systems provide great opportunities for targeted advertisements, by displaying ads alongside genuine recommendations. We consider a biased recommendation system where such ads are displayed without any tags (disguised as genuine recommendations), rendering them indistinguishable to users. We consider the problem of detecting such a bias and propose an algorithm that uses statistical analysis based on binary feedback data from a subset of users. We prove that the proposed algorithm detects bias with high probability for a broad class of recommendation systems with sufficient number of feedback samples.