Without any means of interpretation, the neural network models predicting molecular properties and bioactivities are merely black boxes. We will unravel these black boxes and shed some light into how a neural network can transform a representation of a chemical structure into a biologically meaningful information. We show how single neurons of a neural network can be interpreted as classifiers which determine the presence or absence of pharmacophore-like structures, thereby generating new pharmaceutically relevant knowledge for both pharmacology and biochemistry. We further discuss how these novel toxicophores can be extracted from the network by identifying the most relevant components of a compound for the network's prediction. We envision that having access to such interpretable knowledge is a crucial aid in the development and design of new pharmaceutically active molecules, and helps to investigate and understand failures and successes of current methods.