Recent Papers

August 14, 2023August 17, 2023 Qianxiao Publications and Preprints DD, EAJAM, GSI, ICML, JML, Nat Rev Mater, npj Comp Mat, Papers

Here are some recent published papers grouped roughly in three directions

Approximation Theory for deep learning

A Brief Survey on the Approximation Theory for Sequence Modelling: In this recent review paper, we survey past and present works on the approximation theory of modelling sequential relationships, including those using recurrent networks, convolutional networks, encoder-decoder networks, and the transformers family of architectures. This should serve as a starting point for researchers who are interested in problems on the intersection of approximation theory, nonlinear dynamics and sequence modelling (chatGPT!). This is published in the newly established journal, Journal of Machine Learning.
Forward and Inverse Approximation Theory for Linear Temporal Convolutional Networks: In this paper, we further develop the approximation theory of temporal convolutional architectures, including a refined Jackson-type estimate for the approximation error, as well as a new Bernstein-estimate. This furthers the understanding of the structural bias of temporal (dilated) convolutional networks for sequence modelling. This will be presented as an oral talk at GSI 2023.
Approximation Analysis of Convolutional Neural Networks: In this work, we establish some approximation results for convolutional networks, paying particular attention to the compositional structure and how it can lead to (efficient) approximation. This is published in the East Asian Journal on Applied Mathematics.

Machine learning + scientific computing + control

Principled Acceleration of Iterative Numerical Methods Using Machine Learning: In this paper, we investigate the application of meta-learning type approaches to speed up iterative algorithms in scientific computing. An example is guessing the initial condition for a Jacobi solver (say for the Poisson equation as a sub-step in a Navier-Stokes equation solver). We show that a naive application of meta-learning (MAML algorithm) does not necessarily lead to gains in performance – contrary to what has been suggested in many recent empirical works. We concretely investigate this phenomena through analytical examples, and propose principled solutions to this dilemma. This work is published in ICML 2023.
Fairness In a Non-Stationary Environment From an Optimal Control Perspective: In this work, we connect the fairness problem in machine learning with control theory: in particular, ensuring that a machine learning model is fair to all demographic groups in a changing environment can be understood as an optimal control problem – in particular some sort of stabilisation problem. This allows us to design control strategies to promote fairness dynamically. This work is presented as a workshop paper in ICML 2023.

Machine learning + materials science

Knowledge-integrated machine learning for materials: lessons from gameplaying and robotics: In this perspective paper, we review and envision the fusion of machine learning and materials science research, making parallels to the development of game playing and robotics. This is published in Nature Reviews Materials.
Fast Bayesian optimization of Needle-in-a-Haystack problems using zooming memory-based initialization (ZoMBI): In this paper, we propose a modified Bayesian optimisation method that are shown to be effective for a class of optimisation problems involving the inverse design for materials discovery. This is the full publication corresponding to our earlier neurIPS workshop paper, and is now published in npj Computational Materials.
Tackling data scarcity with transfer learning: a case study of thickness characterization from optical spectra of perovskite thin films: In this paper, we tackle the data-scarcity issue in the application of machine learning in experimental sciences. In particular, we investigate the characterisation of thickness in perovskite thin films using data-driven techniques, yielding a transfer-learning framework that can be used for more general problems. This work is published in Digital Discovery.

Recent Papers

January 8, 2023January 8, 2023 Qianxiao Publications and Preprints ACS, JCP, MSML, NeurIPS, Papers, SISC, TMLR

Here are some recent published papers.

On stability and regularisation for data -driven solution of parabolic inverse source problems: In this paper, we investigate the stability properties on using neural networks for inverse problems involving parabolic equations with source terms, and appropriate regularisation strategies such as gradient penalisation, that improves the solution of these inverse problems. This work is published in the Journal of Computational Physics.
Computing high-dimensional invariant distributions from noisy data: In this paper, we further develop methods to compute invariant distributions of SDEs from noisy sampled data. The key idea is to use a quasi-potential type decomposition of the force field and using neural networks as surrogates for the decomposition. This allows us to compute invariant distributions of high-dimensional systems, especially when the noise is small (so that the distribution becomes quite singular). This work is published in the Journal of Computational Physics.
Accelerating the Discovery of Rare Materials with Bounded Optimization Techniques: In this paper, we propose a modified Bayesian optimisation method that are shown to be effective for a class of optimisation problems involving the inverse design for materials discovery. This work is presented as a spotlight at the NeurIPS 2022 Workshop on AI for Accelerated Materials Design.
Dynamic Modeling of Intrinsic Self-Healing Polymers Using Deep Learning: In this paper, we develop a methodology to learn interpretable dynamics from experimental data on self-healing polymers. We show that even with a small dataset and in the presence of noise, an appropriate physics-based model ansatz combined with machine learning enables prediction of the dynamics of healing, and the study of toughness evolutions as functions of time. We envision this as a first step towards a principled combination of self-healing polymer analysis and design using deep learning. This work is published in ACS Applied Materials and Interfaces.
From Optimization Dynamics to Generalization Bounds via Łojasiewicz Gradient Inequality: In this work, we investigate the connection between optimisation and generalisation in machine learning. The main idea is that one can relate the length of an optimisation path with generalisation estimates. Moreover, a LG type inequality can be used to control the optimisation trajectory length, thereby obtaining generalisation bounds. We show that this method allows for the establishment of a number of generalisation estimates (both known and new) for machine learning models. This work is published in Transactions on Machine Learning Research (TMLR).
Adaptive sampling methods for learning dynamical systems: In this work, we develop numerical methods for sampling trajectories that are used to train dynamical models using machine learning. Unlike static supervised learning problems, here there involves a distribution shift caused by disparity between sampling probability measures and target probability measures. The sampling method proposed in this work makes use of this disparity to design efficient sampling strategies. This work is published in the proceedings of Mathematical and Scientific Machine Learning (MSML).
Personalized Algorithm Generation: A Case Study in Learning ODE Integrators: In this work, we develop a machine learning method for the acceleration of RK-type integrators using ideas from multi-task learning. In particular, we show that if one solves repeatedly problems of a similar type, it is advantageous to adapt the integrator structure (here the RK coefficients) to obtain speedups. This work is published in SIAM Journal on Scientific Computing.

Recent papers

June 19, 2021June 19, 2021 Qianxiao Publications and Preprints Adv Func Mat, CVPR, ICML, npj Comp Mat, Papers

Here are a number of recent accepted/published papers in machine learning and computer vision:

Approximation Theory of Convolutional Architectures for Time Series Modelling: In this paper, we develop some approximation theory for convolutional based architectures for time series analysis – with WaveNet as a prime example. This can be seen as a parallel for the approach taken in our previous paper, but this time for convolutional networks instead of recurrent networks. Our key finding here is that convolutional structures exploits certain “effective low rank” structures for efficient approximation, which can be very different from the “exponentially decaying memory structures” that RNN brings. This paper will appear at ICML 2021.
Adversarial Invariant Learning: In this paper, we develop methods to use adversarially chosen data splits to tackle the out-of-distribution generalization problems. This paper is published at CVPR 2021.

We also have a number of recent papers on the application of machine learning to science and engineering:

A Data Driven Method for Computing Quasipotentials: In this paper, we develop a efficient method for computing the Freidlin-Wentzell quasipotential for rare event analysis, especially in high dimensions. The idea is based on learning a decomposition of the force field, from which the quasipotential can be identified. This will appear in MSML 2021.
Two-step machine learning enables optimized nanoparticle synthesis: In this paper, we develop inverse design methodologies for nanoparticles of desired properties, through a combination of Bayesian optimization and neural network based methods. This paper is published at Npj Computational Materials.
Machine learning and high-throughput robust design of P3HT-CNT composite thin films for high electrical conductivity: In this paper, we investigate how to design carbon nanotubes with desired conductive properties through machine learning. The key difficulty here is the large amount of experimental noise, and we overcome this issue by developing a graph-based regression approach that takes more experimental information into account. This paper will appear in Advanced Functional Materials.

Paper accepted at JEMS

March 21, 2021March 21, 2021 Qianxiao Publications and Preprints Arxiv, JEMS, Papers

Our paper Deep Learning via Dynamical Systems: An Approximation Perspective has been accepted at the Journal of the European Mathematical Society (JEMS).

In this paper, we set up the mathematical foundations of the approximation theory of deep learning idealized as continuous dynamical systems. Our main result is a set of general sufficient conditions on the flow field that implies the universal approximation of such networks. This is a first step towards uncovering the power of composition (which in continuous-time is just dynamics) on approximation of functions.

Recent Preprints

March 16, 2021March 21, 2021 Qianxiao Publications and Preprints Arxiv, Papers

Here are some recent co-authored preprints on the applications of machine learning methods for analysis of dynamical systems (model reduction, rare events)

You can read more about them in my research descriptions (here and here).

Accepted Papers at ICLR and AAAI 2021

March 15, 2021March 21, 2021 Qianxiao Publications and Preprints AAAI, ICLR, Papers

Two co-authored papers have been accepted for publication in ICLR 2021

On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis: In this paper, we formulate the basic mathematical setting of learning dynamical relationships using recurrent neural networks, and uncover precisely the effect of memory on approximation and optimization in the linear setting.
Towards Robust Neural Networks via Close-loop Control: In this paper, we use the optimal control formulation of deep learning to develop adversarial defense mechanisms based on closed-loop control.

A co-authored paper has been accepted at AAAI 2021

Amata: An Annealing Mechanism for Adversarial Training Acceleration: In this paper, we employ control-theoretic approaches to analyze and improve upon classical PGD based methods for adversarial training.

Qianxiao Li (李千骁)

Department of Mathematics & Institute of Functional Intelligent Materials, National University of Singapore

Papers

Recent Papers

Recent Papers

Recent papers

Paper accepted at JEMS

Recent Preprints

Accepted Papers at ICLR and AAAI 2021