Zhenglun Kong

I am currently a postdoctoral researcher at Harvard, working closely with Marinka Zitnik at Harvard and Manolis Kellis at MIT. I received my PhD from Northeastern University in 2024, supervised by Prof. Yanzhi Wang. Prior to that, I earned my master degree from Northeastern University in 2019 and B.E. degree from Huazhong University of Science and Technology (HUST), China, in 2017. I was a research intern at Microsoft Research, ARM, and Samsung Research. My research focuses on developing efficient deep learning methods applicable to real-world scenarios, including computer vision and natural language processing. I was selected as the Machine Learning and Systems Rising Stars 2024.

Email  /  CV  /  Google Scholar  /  Github  /  LinkedIn  /  Twitter

I am dedicated to achieving the general and practical implementation of AI. My research is particularly focused on the following domains and methodologies:

• Efficient Deep Learning: Accelerating pre-training/fine-tuning and inference, data/model compression, and fast & robust DNN design for Large Language Models and Vision Models.

• AI for Health: Efficient algorithms for AI agents and cell foundation models.

• 
  Harvard University, August 2024 - present
  Postdoctoral researcher
  Mentors: Marinka Zitnik, Manolis Kellis
• 
  Microsoft, June 2022 - August 2022
  Research Intern
  Mentor: Subhabrata Mukherjee
• 
  ARM, June 2021 - August 2021
  Research Intern
  Mentors: Lingchuan Meng, Danny Loh
• 
  Samsung Research America, October 2021 - December 2021
  Research Intern
  Mentors: Avik Ray, Yilin Shen, Hongxia Jin

We propose Q-TempFusion, a novel approach for temporal multi-sensor fusion designed to enhance the BEV model’s inference speed while keeping high predictive performance.

We revisit the unique computational characteristics of SSMs and discover that naive application disrupts the sequential token positions. This insight motivates us to design a novel and general token pruning method specifically for SSM-based vision models.

We propose a tailored, unified post-training token reduction method for SSMs. Our approach integrates token importance and similarity, thus taking advantage of both pruning and merging, to devise a fine-grained intra-layer token reduction strategy.

We design an entropy \& distribution guided quantization method to reduce information distortion in quantized query, key, and attention maps, tackling the bottleneck of QAT for LLMs.

We propose an activation-guided quantization framework for popular Large Language Models (LLMs). Specifically, with 4-bit or 8-bit for the activation and 4-bit for the weight quantization.

We introduce a truly efficient, hardware-oriented approach for searching efficient vision transformer structure. This approach has been optimized to seamlessly adapt to the constraints of the target hardware and fulfill the specific speed requirements.

We present a hardware-oriented transformer-based framework for 3D detection tasks, which achieves higher detection precision and remarkable speedup across high-end and low-end GPUs.

We propose a novel speed-aware transformer for end-to-end object detectors, achieving high-speed inference on multiple devices.

We introduce sparsity into data and propose an end-to-end efficient training framework to accelerate ViT training and inference.

We generalize the LTH for ViTs to input data consisting of image patches inspired by the input dependence of ViTs. That is, there exists a subset of input image patches such that a ViT can be trained from scratch by using only this subset of patches and achieve similar accuracy to the ViTs trained by using all image patches.

We propose a dynamic, latency-aware soft token pruning framework for Vision Transformer. Our framework significantly reduces the computation cost of ViTs while maintaining comparable performance on image classification.

We propose an efficient transformer-based large-scale language representation using hardware-friendly block structure pruning. We incorporate the reweighted group Lasso into block-structured pruning for optimization.

We propose a compression-compilation codesign framework that can guarantee BERT model to meet both resource and real-time specifications of mobile devices.

We realize automatic image segmentation with convolutional neural network to extract the accurate contours of four tissues: the skull, cerebrospinal fluid (CSF), grey matter (GM), and white matter (WM) on 5 MRI head image datasets.

• 
  Northeastern University, Sep 2019 - Present,
  PhD in Computer Engineering
  Advisor: Prof. Yanzhi Wang
• 
  Northeastern University, Sep 2017 - May 2019
  M.S. in Computer Engineering
  Advisor: Prof. Yun Raymond Fu
• 
  Huazhong University of Science and Technology, China, Sep 2013 - July 2017
  B.E. in Optoelectronic Information Science and Engineering
  

• Teaching Assistant at Northeastern University

  • EECE 7205 Fundamentals of Computer Engineering, Fall 2021
    Instructor: Prof. Xue Lin   

  • EECE 5552 Assistive Robotics, Fall 2019
    Instructor: Prof. Alireza Ramezani   

  • EECE 5644 34595 Machine Learning/Pattern Recognition, Spring 2019
    Instructor: Prof. Jennifer Dy   

• Guest Lecturer

  • CSC 791&591 Advanced Topics in Efficient Deep Learning
    North Carolina State University, 2022 Fall
    Instructor: Prof. Dongkuan Xu   

• Towards Efficient Deep Learning for Practical AI Implementation
  Carnegie Mellon University, Pittsburgh, PA, Jan. 2024.
  

• The Lottery Ticket Hypothesis for Vision Transformers
  IJCAI, Macao, SAR, Aug. 2023.
  

• Peeling the Onion: Hierarchical Reduction of Data Redundancy for Efficient Vision Transformer Training
  AAAI, Washington, DC, Feb. 2023.
  

• Enabling Faster Vision Transformers via Soft Token Pruning (link)
  The 8th EMC2 - Energy Efficient Training and Inference of Transformer Based Models, Washington, DC, Feb. 2023.
  

• Vision Transformer Optimization
  ARM, San Jose, CA, Aug. 2021.
  

• A Compression-Compilation Framework for On-mobile Real-time BERT Applications.
  IJCAI, Montreal-themed virtual reality, Aug 2021.
  

• Hardware-friendly Block Structured Pruning for Transformer
  ARM, San Jose, CA, Jun. 2021.
  

• Compiler-aware Neural Architecture Optimization for Transformer
  Samsung Research America, Mountain View, CA, Oct. 2020.
  

• Conference Reviewer:
  • ICML'22, ECCV'22, NeurIPS'22, AAAI'23, CVPR'23, KDD'23, IJCAI'23, ICML'23, NeurIPS'23, AAAI'23, ICML'24, ECCV'24, NeurIPS'24, EMNLP'24, WACV'25, ICLR'25

• Journal Reviewer:
  • IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI)
  • IEEE Transactions on Neural Networks and Learning Systems (TNNLS)
  • IEEE Transactions on Image Processing (TIP)
  • Pattern Recognition
  • Neurocomputing

• Academic Committee Member:
  • MLNLP