Hi! I’m Tianle Cai (蔡天乐, pronounced as Tyen-luh Tseye), a Ph.D. student at Princeton University. Before joining Princeton, I studied at Peking University (PKU) majoring in applied mathematics, while pursuing a double major in computer science. I was fortunate to be advised by Professor Liwei Wang on research about theory of machine learning. I spent a wonderful summer at MIT as a research intern supervised by Professor Sasha Rakhlin in 2019. I also work closely with Professor Jason D. Lee and Doctor Di He.

I have a very broad range of interests spanning many fields in machine learning, e.g., optimization, representation learning, architecture design (Transformer, Graph Neural Networks, etc.). To summarize in one sentence, I’m mostly interested in the theories that can inspire us to make better algorithms. My tenet is to make machine learning more general [1, 2, 4], efficient [3, 6, 7, 10], and reliable [5, 8, 9] (please see my publications below).

If you are interested in collaborating with me or want to have a chat, always feel free to contact me through e-mail or WeChat : )

News

• Three papers accepted by NeurIPS 2021! Sep. 2021
• Graphormer wins first place in PCQM4M task of OGB-LSC @ KDD Cup 2021! June, 2021
• Three papers accepted by ICML 2021! May, 2021

Selected Publications

1. (NeurIPS 2021) Do Transformers Really Perform Bad for Graph Representation?

   Chengxuan Ying, Tianle Cai, Shengjie Luo, Shuxin Zheng, Guolin Ke, Di He, Yanming Shen, Tie-Yan Liu

   

   Highlight: Make Transformer great again on graph classification by introducing three graph structural encodings! Achieve SOTA performance on several benchmarks! Winner solution of OGB-LSC challenge!!

   [Code]

2. (NeurIPS 2021) Towards a Theoretical Framework of Out-of-Distribution Generalization

   Haotian Ye, Chuanlong Xie, Tianle Cai, Ruichen Li, Zhenguo Li, Liwei Wang

   

   Highlight: We formulate what an OOD is and derive bounds and model selection algorithm upon our framework.

3. (NeurIPS 2021) Stable, Fast and Accurate: Kernelized Attention with Relative Positional Encoding

   Shengjie Luo, Shanda Li, Tianle Cai, Di He, Dinglan Peng, Shuxin Zheng, Guolin Ke, Liwei Wang, Tie-Yan Liu

   

   Highlight: Enabling fast relative positional encoding and stabilize the training via Fast Fourier Transform.

4. (ICML 2021) A Theory of Label Propagation for Subpopulation Shift

   Tianle Cai*, Ruiqi Gao*, Jason D. Lee*, Qi Lei*

   

   Highlight: Subpopulation shift is a ubiquitous component of natural distribution shift. We propose a general theoretical framework of learning under subpopulation shift based on label propagation. And our insights can help to improve domain adaptation algorithms.

   [Slides]

5. (ICML 2021) Towards Certifying $\ell_\infty$ Robustness using Neural Networks with $\ell_\infty$-dist Neurons

   Bohang Zhang, Tianle Cai, Zhou Lu, Di He, Liwei Wang

   

   Highlight: New architecture with inherent $\ell_\infty$-robustness and a tailored training pipeline. Achieving SOTA performance on several benchmarks!

   [Code]

6. (ICML 2021) GraphNorm: A Principled Approach to Accelerating Graph Neural Network Training

   Tianle Cai*, Shengjie Luo*, Keyulu Xu, Di He, Tie-Yan Liu, Liwei Wang

   

   Highlight: A principled normalization scheme specially designed for graph neural networks. Achieve SOTA on several graph classification benchmarks.

   [Code], [Third-part implementation by microsoft ptgnn lab. (Thanks for the very quick reaction and implementation MS!)], [Slides]

7. (NeurIPS 2020) Sanity-Checking Pruning Methods: Random Tickets can Win the Jackpot

   Jingtong Su*, Yihang Chen*, Tianle Cai*, Tianhao Wu, Ruiqi Gao, Liwei Wang, Jason D. Lee

   

   Highlight: We sanity-check several existing pruning methods and find the performance of a large group of methods only rely on the pruning ratio of each layer. This finding inspires us to design an efficient data-independent, training-free pruning method as a byproduct.

   [Code]

8. (NeurIPS 2020) Locally Differentially Private (Contextual) Bandits Learning

   Kai Zheng, Tianle Cai, Weiran Huang, Zhenguo Li, Liwei Wang

   

   Highlight: Simple black-box reduction framework improves private bandits bounds.

   [Code]

9. (NeurIPS 2019 Spotlight 2.4 % Acceptance rate) Convergence of Adversarial Training in Overparametrized Networks

   Ruiqi Gao*, Tianle Cai*, Haochuan Li, Liwei Wang, Cho-Jui Hsieh, Jason D. Lee

   

   Highlight: For overparameterized neural network, we prove that adversarial training can converge to global minima (with loss 0).

   [Slides]

10. (NeurIPS 2019 Beyond First Order Method in ML Workshop) Gram-Gauss-Newton Method: Learning Overparameterized Neural Networks for Regression Problems

    Tianle Cai*, Ruiqi Gao*, Jikai Hou*, Siyu Chen, Dong Wang, Di He, Zhihua Zhang, Liwei Wang

    

    Highlight: A provable second-order optimization method for overparameterized network on regression problem! As light as SGD at each iteration but converge much faster than SGD for real world application.

Talks

• Label Propagation for Domain Adaptation at AI TIME [slides]

• Towards Understanding Optimization of Deep Learning at IJTCS [slides] [video]

• A Gram-Gauss-Newton Method Learning Overparameterized Deep Neural Networks for Regression Problems at PKU machine learning workshop [slides]

Experience

• Visiting Research Student at Simons Institute, UC Berkeley
  • Program: Foundations of Deep Learning
  • June, 2019 - July, 2019
• Visiting Research Internship at MIT
  • Advisor: Professor Sasha Rakhlin
  • June, 2019 - Sept., 2019
• Visiting Research Student at Princeton
  • Host: Professor Jason D. Lee
  • Sept., 2019 - Oct., 2019