Representation learning on graphs enables downstream tasks such as node/graph classification and link prediction in multiple application scenarios. We are developing novel methods for analyzing graph representations and their relationship to the Graph Neural Networks (GNNs) generating them and to the downstream tasks. We are interested in manifold analysis for quantifying the similarity between two representation spaces and robust means for comparing and aligning representations.