I worked with two teammates to investigate the depth properties of Deep Equilibrium Models (DEQs), a new class of implicit DL sequential models that mimic "infinite" layer neural networks. I had zero prior ML experience and completed this initial research project in only 7 weeks, including literature review, learning PyTorch, Google Cloud Compute, Git, and more. The task I designed involved classifying symbolic arithmetic expressions allowed us to test the depth properties of DEQs.  Our project was awarded "Best Project" in the hallmark graduate level course (my team was all freshmen), demonstrating my proficiency in DL sequential models and ability to learn, pick up, and creatively manipulate dense technical concepts and tools.

Unlike traditional neural networks, which stack many layers of weights and nonlinearities, DEQs model the solution of a fixed-point equation that is a continuous analogue of residual networks. In other words, DEQs models a network where a “layer” is applied countless times until convergence, and thus an “infinite” layer network. These models have the potential to outperform traditional networks on tasks that require a deeper architecture, but most recent works have been with larger scale applications. Due to the compute limitations, we had to get creative to design a smaller scale experiment that still highlighted DEQs advantages and would allow us to explore further.