With the introduction of BERT, we suddenly have a strong performing, generalizable model that can be transferred to a variety of tasks. But, BERT is really, really large. During this talk, we will explore how to reduce the size of BERT while retaining its capacity in the context of Question Answering tasks. We will pair distillation with Multimetric Bayesian Optimization. By concurrently tuning metrics like model accuracy and number of model parameters, we will be able to distill BERT and assess the trade-offs between model size and performance. This experiment is designed to address two questions through this process:
By combining distillation and Multimetric Bayesian Optimization, can we better understand the effects of compression and architecture decisions on model performance? Do these architectural decisions (including model size) or distillation properties dominate the trade-offs?
Can we leverage these trade-offs to find models that lend themselves well to application specific systems (ex: productionalization, edge computing, etc)?