How Powerful are Graph Neural Networks Jure Leskovec Tasks on Networks Classical ML tasks in networks: § Node classification § Predict a type of a given node § Link prediction § Predict whether two nodes are linked § Community detection § Identify densely linked clusters of nodes § Network similarity § How similar are two (sub)networks Jure Leskovec, Stanford University 2 ML on Graphs ? ? ? ? ? Machine Learning Example: Node Classification Many possible ways to create node features: § Node degree, PageRank score, Motifs, Degree of neighbors, Clustering, … Jure Leskovec, Stanford University 3 Deep Learning in Graphs … z Input: Network Predictions: Node labels, New links, Generated graphs and subgraphs Jure Leskovec, Stanford University 4 Why is it Hard? § Modern deep learning toolbox is designed for simple sequences or grids § CNNs for fixed-size images/grids…. § RNNs or word2vec for text/sequences… Jure Leskovec, Stanford University 5 Why is it Hard? But networks are far more complex! § Arbitrary size and complex topological structure (i.e., no spatial locality like grids) vs. Text Networks Images § No fixed node ordering or reference point § Often dynamic and have multimodal features Jure Leskovec, Stanford University 6 Setup We have a graph !: § " is the vertex set § # is the (binary) adjacency matrix '×|*| § $∈ℝ is a matrix of node features § Meaningful node features: § Social networks: User profile § Biological networks: Gene expression profiles, gene functional information Jure Leskovec, Stanford University 7 Graph Neural Networks Idea: Node’s neighborhood defines a computation graph ! Determine node computation graph ! Propagate and transform information Learn how to propagate information across the graph to compute node features The Graph Neural Network Model. Scarselli et al. IEEE Transactions on Neural Networks 2005 Semi-Supervised Classification with Graph ConvolutionalJure Networks. N. Kipf,University M. Welling, ICLR 2017 Leskovec,T.Stanford 8 Graph Neural Networks A C B B TARGET NODE A A C B A C E F D F E D INPUT GRAPH A Each node defines a computation graph § Each edge in this graph is a transformation/aggregation function Inductive Representation Learning on Large Graphs. W. Hamilton, R. Ying, J. Leskovec. NIPS, 2017. Jure Leskovec, Stanford University 9 Graph Neural Networks A C B B TARGET NODE A A C B A C E F D F E D A INPUT GRAPH Neural networks Intuition: Nodes aggregate information from their neighbors using neural networks Inductive Representation Learning on Large Graphs. W. Hamilton, R. Ying, J. Leskovec. NIPS, 2017. Jure Leskovec, Stanford University 10 Idea: Aggregate Neighbors Intuition: Network neighborhood defines a computation graph Every node defines a computation graph based on its neighborhood! Can be viewed as learning a generic linear combination of graph low-pass and high-pass operators Jure Leskovec, Stanford University 11 [NIPS ‘17] Our Approach: GraphSAGE !( #' ∈ ) *ℎ% ) xA xC >… combine ?… aggregate ? > Update for node +: (-./) ℎ, 1<= =1 2 8+ level embedding of node * - # ℎ,- , 4 %∈) , Transform *’s own embedding from level 8 1(5 - ℎ%- ) Transform and aggregate embeddings of neighbors 9 6 § ℎ, = attributes 7, of node * Jure Leskovec, Stanford University 12 [NIPS ‘17] GraphSAGE: Training § Aggregation parameters are shared for all nodes § The number of model parameters is sublinear in |V| § Can use different loss functions: * § Classification/Regression: ℒ ℎ% = '% − ) ℎ% § Pairwise Loss: ℒ ℎ% , ℎ, = max(0, 1 − 3456 ℎ% , ℎ, ) shared parameters B A 8 (9) , : (9) C F D shared parameters E INPUT GRAPH Compute graph for node A Jure Leskovec, Stanford University Compute graph for node B 13 [NeurIPS ‘18] Pooling for GNNs Don’t just embed individual nodes. Embed the entire graph. Problem: Learn how to hierarchical pool the nodes to embed the entire graph Our solution: DIFFPOOL § Learns hierarchical pooling strategy § Sets of nodes are pooled hierarchically § Soft assignment of nodes to next-level nodes Hierarchical Graph Representation LearningJure with Differentiable Pooling. R. Ying, et al. NeurIPS 2018. Leskovec, Stanford University 14 DIFFPOOL Architecture A different GNN is learned at every level of abstraction Our approach: Use two sets of GNNs § GNN1 to learn how to pool the network § Learn cluster assignment matrix § GNN2 to learn the node embeddings Hierarchical Graph Representation Learning with Differentiable Pooling. R. Ying, et al. NeurIPS 2018. Jure Leskovec, Stanford University 15 DIFFPOOL Architecture A different GNN is learned at every level of abstraction Next: Many successful extensions and domain Our approach: Use two sets of GNNs adaptations § GNN1 to learn how to pool the network § Learn cluster assignment matrix § GNN2 to learn the node embeddings Hierarchical Graph Representation Learning with Differentiable Pooling. R. Ying, et al. NeurIPS 2018. Jure Leskovec, Stanford University 16 Application (1): Pinterest Task: Recommend related pins tio n An no ta Vi su al N C G N § Challenges: 0.4 0.3 0.2 0.1 0 RW -G C Source pin Mean Reciprocal Rank Task: Learn node embeddings !" s.t. # !$%&'(, !$%&'* < #(!$%&'(, !-.'%/'0 ) § Massive size: 3 billion nodes, 20 billion edges § Heterogeneous data: Rich image and text features Graph Convolutional Neural Networks for Web-Scale Recommender Systems. R. Ying, et al. KDD, 2018. 8 (2) Drug Side Effects § Task: Given a pair of drugs predict adverse side effects 46% of people ages 70-79 take >5 drugs Proteins Drugs § Link prediction on a multimodal graph 36% improvement in [email protected] over state of the art Modeling Polypharmacy Side Effects with GraphJureConvolutional Networks. M. Zitnik, et al. Bioinformatics, 2018.18 Leskovec, Stanford University [NeurIPS ‘18] (3) Targeted Molecule Generation Goal: Generate molecules that optimize a given property (Quant. energy, solubility) Solution: Combination of § Graph representation learning § Adversarial training § Reinforcement learning Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. Jure Leskovec, Stanford University J. You, et al., NeurIPS 2018. 19 [NeurIPS ‘18] (4) Knowledge Graphs Embedding Logical Queries on Knowledge Graphs. W. Hamilton, et al. NeurIPS, 2018. Jure Leskovec, Stanford University 20 How expressive are GNNs? Theoretical framework: Characterize GNN’s discriminative power by analyzing the discriminative power of the multiset function: § Characterize upper bound of the discriminative power of GNNs § Propose a maximally powerful GNN § Characterize discriminative power of popular GNNs that use non-injective multiset functions How Powerful are Graph Neural Networks? K. Xu, et al. ICLR 2019. Jure Leskovec, Stanford University 21 Discriminative Power of GNNs !( #' ∈ )(*)ℎ% ) +… combine ,… aggregate Theorem: The most discriminative GNN uses injective multiset function for neighbor aggregation If the aggregation function is injective, GNN can fully capture/distinguish the rooted subtree structures: Injective Jure Leskovec, Stanford University 22 Discriminative Power of GNNs Theorem: GNNs can be at most as powerful as the Weisfeiler-Lehman graph isomorphism test (a.k.a. canonical labeling or color refinement) What GNN achieves this upper bound? Jure Leskovec, Stanford University 23 Injective functions over multiset Idea: If GNN functions are injective, GNN can capture/distinguish the rooted subtree structures Theorem: Any injective multiset function can be expressed as ! ∑#∈% &(() § Solution: Model !() and &() with MLP Consequence: Sum aggregator is the most expressive How Powerful are Graph Neural Networks? K. Xu, et al. ICLR 2019. Jure Leskovec, Stanford University 24 Expressive Power of GNNs Many popular aggregators: Under review as a conference paper at ICLR 2019 Under review as a conference paper at ICLR 2019 (MEAN {f (x) : x 2 S}) cannot distinguish: A >vs.> A > > Theorem (Informal): sum - multiset mean - distribution max - set sum - multiset mean - distribution max - se Input Mean aggregatorInput only captures distribution information Figure 2: Ranking by expressive power for sum, mean and max-pooling aggregators over a multis der review Under as a conference paper at ICLR 2019 Figure Ranking by expressive power for sum, mean and max-pooling review as aLeft conference paper ICLR 2019and panel2:shows theat input multiset the three panels illustrate the aspects of aggregators the multiset aover give Left panel shows the input multiset and the three panels illustrate the aspects of the multi is able to capture: sum captures the full multiset, mean captures the proportion/distributio (MAX {f aggregator (x) : x S}) aggregator totype, capture: summax captures the full multiset, mean captures thethe proportion/d cannot distinguish: of elements ofis2 aable given and the aggregator ignores multiplicities (reduces multiset to of elements simple set). of a given type, and the max aggregator ignores multiplicities (reduces the m simple set). > >A > > vs. A vs. vs. Theorem (Informal): vs. sum multiset mean distribution max set Input vs. vs.- multiset vs. sum mean - distribution max - set Input Max aggregator only captures set information (no multiplicity) and Maxpower both fail Max fails (c) Mean and Max both fai Figure 2: Ranking (a) byMean expressive for sum, mean and(b)max-pooling aggregators over a multiset. Jure Leskovec, Stanford University 25 > > Power of Aggregators Input sum - multiset mean - distribution max - set Figure 2: Ranking by expressive power for sum, mean and max-pooling aggregators over a multiset. Left panel shows the input multiset and the three panels illustrate the aspects of the multiset a given aggregator is able to capture: sum captures the full multiset, mean captures the proportion/distribution of elements of a given type, and the max aggregator ignores multiplicities (reduces the multiset to a simple set). Failure cases for mean and max agg. A vs. A (a) Mean and Max both fail A vs. A (b) Max fails A vs. A (c) Mean and Max both fail Under Figure review 3: as Examples a conference paper atgraph ICLR 2019 that mean and max-pooling aggregators fail to of simple structures Ranking by discriminative power distinguish. Figure 2 gives reasoning about how different aggregators “compress” different graph structures/multisets. existing GNNs instead use a 1-layer perceptron W (Duvenaud et al., 2015; Kipf & Welling, 2017; A Zhang etAal., 2018), a linear mapping followed by a non-linear activation function such A A as a ReLU. Such 1-layer mappings are examples of Generalized Linear Models (Nelder & Wedderburn, 1972). Therefore, we are interested in understanding whether 1-layer perceptrons are enough for graph learning. Lemma 7 suggests that there are indeed network neighborhoods (multisets) that models sumdistinguish. - multiset mean - distribution max - set Inputperceptrons can never with 1-layer > Jure Leskovec, Stanford University > 26 Summary § Graph Convolution Networks § Generalize beyond simple convolutions § Fuses node features & graph info § State-of-the-art accuracy for node classification and link prediction. § Model size independent of graph size; can scale to billions of nodes § Largest embedding to date (3B nodes, 17B edges) § Leads to significant performance gains Jure Leskovec, Stanford University 27 Conclusion Results from the past 2-3 years have shown: § Representation learning paradigm can be extended to graphs § No feature engineering necessary § Can effectively combine node attribute data with the network information § State-of-the-art results in a number of domains/tasks § Use end-to-end training instead of multi-stage approaches for better performance Jure Leskovec, Stanford University 28 Future Work § Understand other/new GNN variants § Understand optimization and generalization of GNNs § Towards more powerful DL on graphs § GNNs can only distinguish rooted trees: Jure Leskovec, Stanford University 29 References § Tutorial on Representation Learning on Networks at WWW 2018 http://snap.stanford.edu/proj/embeddingswww/ § Inductive Representation Learning on Large Graphs. W. Hamilton, R. Ying, J. Leskovec. NIPS 2017. § Representation Learning on Graphs: Methods and Applications. W. Hamilton, R. Ying, J. Leskovec. IEEE Data Engineering Bulletin, 2017. § Graph Convolutional Neural Networks for Web-Scale Recommender Systems. R. Ying, R. He, K. Chen, P. Eksombatchai, W. L. Hamilton, J. Leskovec. KDD, 2018. § Modeling Polypharmacy Side Effects with Graph Convolutional Networks. M. Zitnik, M. Agrawal, J. Leskovec. Bioinformatics, 2018. § Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generation. J. You, B. Liu, R. Ying, V. Pande, J. Leskovec, NeurIPS 2018. § Embedding Logical Queries on Knowledge Graphs. W. Hamilton, P. Bajaj, M. Zitnik, D. Jurafsky, J. Leskovec. NeuIPS, 2018. § How Powerful are Graph Neural Networks? K. Xu, W. Hu, J. Leskovec, S. Jegelka. ICLR 2019. § Code: § § § § § http://snap.stanford.edu/graphsage http://snap.stanford.edu/decagon/ https://github.com/bowenliu16/rl_graph_generation https://github.com/williamleif/graphqembed https://github.com/snap-stanford/GraphRNN Jure Leskovec, Stanford University 30 Industry Partnerships PhD Students Alexandra Porter Camilo Ruiz Claire Donnat Emma Pierson Funding Jiaxuan You Mitchell Gordon Mohit Tiwari Rex Ying Post-Doctoral Fellows Collaborators Baharan Marinka Mirzasoleiman Zitnik Michele Catasta Robert Palovics Adrijan Bradaschia Rok Sosic Research Staff Srijan Kumar Dan Jurafsky, Linguistics, Stanford University Christian Danescu-Miculescu-Mizil, Information Science, Cornell University Stephen Boyd, Electrical Engineering, Stanford University David Gleich, Computer Science, Purdue University VS Subrahmanian, Computer Science, University of Maryland Sarah Kunz, Medicine, Harvard University Russ Altman, Medicine, Stanford University Jochen Profit, Medicine, Stanford University Eric Horvitz, Microsoft Research Jon Kleinberg, Computer Science, Cornell University Sendhill Mullainathan, Economics, Harvard University Scott Delp, Bioengineering, Stanford University Jens Ludwig, Harris Public Policy, University of Chicago Jure Leskovec, Stanford University 31