MLconf - The Machine Learning Conference

MLconf gathers the machine learning community to discuss recent trends, innovations, research and the application of algorithms, tools, best practices, and platforms within AI/ML.

Posts Filed Under:

Machine Learning

Use Transfer Learning for Efficient Deep Learning Training

One of the biggest challenges with deep learning is the large number of labeled data points that are required to train deep learning models to sufficient accuracy. For example, the ImageNet*1 database for image recognition consists of over 14 million hand-labeled images. While the number of possible applications of deep learning systems in vision tasks, text processing, speech-to-text translation and many other domains is enormous, very few potential users of deep learning systems have sufficient training data to create models from scratch. A common concern among teams considering the use of deep learning to solve business problems is the need for training data: “Doesn’t deep learning need millions of samples and months of training to get good results?” One powerful solution is transfer learning, in which part of an existing deep learning model is re-optimized on a small data set to solve a related, but new, problem. In fact, one of the great attractions of transfer learning is that, unlike most traditional approaches to machine learning, we can take models trained on one (perhaps very large) dataset and modify them quickly and easily to work well on a new problem (where perhaps we have only a very small dataset). Transfer learning methods are not only parsimonious in their training data requirements, but they run efficiently on the same CPU-based systems that are widely used for other analytics workloads, including machine learning and deep learning inference.

Transfer Learning Basics

The idea of transfer learning is inspired by the fact that people can intelligently apply knowledge learned previously to solve new problems. For example, learning to play one instrument can facilitate faster learning of another instrument. Another good analogy is with traditional software development: We almost never write a program completely from scratch; every application makes heavy use of code libraries that take care of common functionality. Maximizing code reuse is a best practice for software development, and transfer learning is essentially the machine learning equivalent.

As described in this article Start your data analytics journey today with open source and transfer learning;2 transfer learning is an artificial intelligence (AI) practice that uses data, deep learning recipes, and models developed for one task, and reapplies them to a different, but similar, task. In other words, it’s a method in machine learning where a model developed for one task is used as a starting point for a model in a second task. Reuse of pre-trained models allows improved performance when modeling the second task, hence achieving results faster.

Vast quantities of readily available data are great, but it isn’t a prerequisite for success. With modern machine learning and deep learning techniques, knowledge acquired by a machine working on one task can be transferred to a new task if the two are somewhat related. This eventually helps to reduce training time significantly, thus improving productivity of data scientists. There are three main questions to consider when trying to implement transfer learning: What to transfer, how to transfer and when to transfer. What to transfer asks which part of knowledge can be transferred across domains. Once what to transfer has been addressed the next step is to develop algorithms and models to transfer the knowledge; this falls under the how to transfer step. And the last question asks to which cases should the knowledge be transferred and more importantly not be transferred. In certain situations, brute transfer may even hurt performance of the target task, often referred to as negative transfer.

Now, that we understand some basics of transfer learning, let’s look at few use cases of transfer learning. But before we do that, let me remind you that we do not always need enormous amounts of data for end-end deep learning training. With use cases like facial recognition, where the fundamental features used for classification don’t change, there is no need to retrain the complete deep neural network. Transfer learning can be employed in such scenarios, where the features learned using a large dataset are transferred to the new network and only the classifier part is trained with the new, much smaller dataset, as shown in Figure 1.


Transfer Learning in Practice

Transfer learning has been applied in numerous places to solve variety of real world problems. One such unique example is use of transfer learning to find missing children. More than 465,676 missing children were reported to the Federal Bureau of Investigation in 2016 alone12. More than 100,000 escort advertisements are posted online every day, and one in six children reported missing is a possible victim of sex trafficking, as reported by the National Center for Missing and Exploited Children. Intel has worked with Thorn4 to address the challenge of matching the images of children in the online escort ads with the pictures of known missing children. Thorn is an organization that was able to leverage technology to fight child sex trafficking and apply transfer learning to tackle their huge data challenge.1,3 Intel helped Thorn take open source models trained on general images of adults and reuse the system to recognize and match images of trafficking victims. To further improve the ability of Thorn to find trafficking victims, Intel used transfer learning on Intel Xeon processors to retrain the model. Using a small dataset of a thousand victims, they took what the algorithm could already do, match general images of adults, and repurpose it to apply it to the new problem.

Another area where transfer learning is gaining popularity is medical image analysis. There have been several research publications5,6,7 in the past few years on how transfer learning is being employed to detect diseases on medical images with a high level of accuracy. A few researchers have demonstrated the application of transfer learning to detect eye diseases in medical images, with accuracy comparable to human experts. They have applied transfer learning to accelerate the diagnosis of age-related macular degeneration (AMD) in medical images of the eye. What’s remarkable about this, and other application of transfer learning in medical image analysis, is that it will lead to expediting the diagnosis and referral of treatable medical conditions, resulting in early treatment and improved clinical outcomes.

There is no dearth of use cases where transfer learning can be applied with a high level of resultant accuracy. Apart from the application of transfer learning in medical image analysis, there have been studies around facial verification10, sentiment analysis8 as well as in mispronunciation detection9. The fact that transfer learning doesn’t need a huge amount of data for training and can repurpose extracted features from a different source task makes it a great technique to apply and use in a variety of domains.

Getting Started on AI with Transfer Learning

AI has the potential to revolutionize the world. There has been significant growth in the number of application domains where AI can be applied. There are a variety of ways to get started on AI, and this blog describes one of them, transfer learning. Transfer learning eliminates the need for specialized hardware and large datasets, making it easier and faster for users to deploy AI workloads. By using transfer learning, developers can use their current infrastructure with a limited amount of data and start their AI journey today. We expect transfer learning to be applicable to various domains where the learned features do not change (thanks to the rules of nature) and can be reused across domains and problems. In the future, transfer learning techniques will potentially be applied to video classification, social network analysis, and logical inference.

We encourage the readers to further explore the applicability and benefits of transfer learning. For those interested in learning more can check out our comprehensive whitepaper on the topic, here11.

References:

  1. http://www.image-net.org/
  2. https://itpeernetwork.intel.com/transfer-learning-data-analytics-ai/
  3. https://software.intel.com/en-us/articles/finding-missing-kids-through-code
  4. https://www.wearethorn.org/
  5. http://www.cell.com/cell/fulltext/S0092-8674(18)30154-5
  6. https://www.sciencedirect.com/science/article/pii/S0010482517300240
  7. http://infolab.stanford.edu/~echang/HTC_OM_Final.pdf
  8. https://dl.acm.org/citation.cfm?id=2020438
  9. https://www.sciencedirect.com/science/article/pii/S0167639314001010
  10. http://openaccess.thecvf.com/content_iccv_2013/papers/Cao_A_Practical_Transfer_2013_ICCV_paper.pdf
  11. https://software.intel.com/en-us/articles/use-transfer-learning-for-efficient-deep-learning-training-on-intel-xeon-processors
  12. http://www.missingkids.com/content/dam/ncmec/en_us/2016%20Annual%20Report.pdf

 

Genomic Precision Medicine as a Small Data Machine Learning Problem

What is this blog about?

If you don’t want to read the whole thing, here is the gist:

  • Precision Medicine (PM) is, to a large extent, a well-established idea of selecting medical treatments based on the patient’s genetics
  • Major (though not all) genomic PM applications can be formulated as Machine Learning (ML) problems
  • These problems are commonly considered to be Big Data problems
  • I will argue that they are in fact Small Data problems
  • This has major implications for methods and software used to develop the PM applications and services for clinical use
  • Smallness of data for PM is a fundamental problem, not technological obstacle

What is Precision Medicine?

PM helps select medical treatments on the basis of certain special characteristics of patient. Of course, medicine has always considered personal characteristics of a patient before determining treatment; the novelty of PM is that it uses patient characteristics, such as genome data, which became available in modern times. A typical scenario is this: you see a doctor for a problem, and she orders a test which “measures” (reads) your genome. The readout is then analyzed by an algorithm, which produces a test report, which may say something like: given your genomic profile, you are likely to benefit from Drug D (Fig. 1). Based on this, and other information, and their expertise, the physician prescribes a treatment. The idea is that physician can make better/more informed choice with the help of the test report. There are lots of variations on this theme, but this is the essence.

Fig. 1. Genome-based personalized medicine.

For completeness: the PM algorithms may consider other types of information, other than genomic. Two prominent examples are: 1) imaging (X-rays, CT scans, retinal scans, etc.) 2) Electronic Health Records (HER). For this blog, I will focus on genomic data, for these reasons:

  • As of 2018, it is still the principal approach to PM
  • It is my area of expertise
  • Imaging and EHR-based PM are substantially different problems. The arguments I put forward for genomic PM may or may not extend to these newer areas

 

Precision Medicine as Machine Learning Problem

In principle, it is clear how PM can be approached as an ML problem. Consider patients suffering from some disease, and split them in two groups: those who have benefited from Drug D, and those who have not (how to define “benefit” is a separate and complicated question… let’s just say there is a binary indicator of benefit called Y). Each patient also has genomic measurement X of some sort, which can be expressed as fixed-length numeric vector. Given this setup, one could apply ML to develop a classifier which assigns the patients into the two groups, responders (those who benefit) and non-responders (those who don’t), using X as predictor variables. Once the classifier is tuned to sufficient accuracy, it can be used to estimate probability of benefit for future patients, based on their genomic readout. Our discussion is whether the problem as described is “Big Data” or not.

For completeness, a note: there is also a PM approach where X is a scalar, not vector. In other words, you measure a single genetic characteristic – for example, mutation or other alteration in a single gene – and derive treatment guidance based on presence or absence of that feature. This method is particularly widespread in oncology, and does not involve ML since it is typically based on biology, i.e., learning from “first principles”. It also has severe limitations, which I touch upon in the last section. In any case, we do not consider that case; in our setup X has measurements on multiple genes or other genomic characteristics.

Precision Medicine as Big Data Problem

First, a clarification: I focus on sample size in this post, not number of features. Thus, by Big Data, I mean lots of samples, and conversely, by Small Data, I mean few samples. The reason is that modern machine learning has effective methods of analyzing data with many features, whereas that is not the case for dealing with small sample sizes. In other words, it is sample size which really matters most, and that’s what I discuss here.

With this background, I’ll argue that the type of PM defined earlier is generally considered to be a Big Data problem. The reason for this is that in recent years there has been an extraordinary exponential growth of the genome (“sequencing”) information generated. Talks in this field almost universally display a graph of number of genomic sequences or nucleotide bases in public databanks as a function of time, pointing to exponential growth. A typical figure is shown below.


Fig. 2. Cumulative number of human genomes sequenced worldwide. From Stephens et al., Big Data: Astronomical or Genomical. PLOS Biology, 2015 (<https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1002195>)

The immense numbers and growth of the reported sequencing data strongly suggest that any ML application purporting to produce PM algorithms (classifiers) must be dealing with Big Data. If true, that would be a good thing, because given that much data we ought to be able to develop very accurate classifiers, and thereby help many patients.

However, the argument that PM is a Big Data problem has a flaw. First, consider some fairly obvious background.
To derive genomic classifier for typical PM application, one has to train ML algorithms using predictor variables X (i.e., the genome data) and dependent variable Y (i.e., “phenotypic” data). In other words, for each “training set” patient there must be X and Y available. The Y data depends on the disease of interest. Some types of binary Y information are: who will experience a particular disease outcome (for example, worsening of disease, cancer recurrence, death) in a given timeframe (this is called prognosis); which patients from particular disease benefit from a given drug (this is called prediction; for example antibiotic resistance); and which patients have a particular disease or disease type (for example, bacterial vs. viral infection), given a genomic signature found in their body (blood, tumor tissue, nasal swab etc.; this is called diagnosis).

Given this background, the key problem with the Big Data argument is that the growth of Y data lags far behind the genomic (X) data. You will seldom, if ever, see a chart similar to growth of sequencing data, but illustrating the growth of the phenotypic data. And yet, it is Y that determines what data is available for training PM algorithms, because, as we’ve seen, we must have X and Y for each patient. In fact, my experience and large published literature show that the amount of Y data for almost any specific clinical problem of interest is severely limited. There are many reasons, but a critical one is time: it can take a long time to acquire sufficient numbers of Y (dependent variable) data.

Consider, for example, a hugely successful application of PM in guiding the chemotherapy treatment for early-stage breast cancer patients (<https://tinyurl.com/ybfgdaxq>). In this case, a classifier was trained to predict whether the patient’s cancer will recur within 10 years of diagnosis (recurrence). If the cancer is likely to recur, chemotherapy can be given to reduce the chances of that happening. On the other hand, vast majority of patients do not need the toxic chemo because their cancers are not coming back. Therefore it is of great interest to predict recurrence, and limit chemo to just those patients likely to recur. To train the model to predict recurrence, as usual two sources of data must be available:

  • The genomic data X, which is more or less readily available in case of breast cancer
  • The 10-year recurrence information: Y = 1 if cancer recurred, 0 otherwise. Depending on the type of ML model, time of recurrence is also recorded.

 

In other words, we must wait full 10 years for each patient, to learn their Y variable (and time, if used). In practice, the wait is longer, because we can’t recruit the full cohort of patients on Day 1. Needless to say, this imposes severe limitations on the development of this type of tests, because few businesses (and investors) can afford/are willing to wait this long for data. In fact, it is remarkable that this particular application has been developed at all. Worst of all, very little can be done to alleviate this problem – no amount of investment or technological progress can fundamentally alter the fact that we have to wait for outcomes to happen.

Given this situation, we see that the sample size for genomic PM is truly driven by the amount of outcome (Y) data, not by X. And given the times it takes to acquire typical outcomes, the numbers commonly available to PM algorithm developers commonly number in the hundreds, if not less. I experienced this first hand, having spent 15 years developing genomic PM applications, and being always short on data, without exception. For this reason typical PM ML applications are in fact Small Data problems. And for this reason, such applications are not very widespread, as evidenced by the fact that there are only a handful such products in clinical use.

Precision Medicine as Small Data Problem

Given the above reasoning, we argued that ML-based genomic PM applications are Small Data problems. This brings up two more questions. 1) what exactly is “number of samples”? what does “Small” actually mean (i.e., how many samples constitute Small Data).

Let’s discuss 1) first. This sounds like a very simple question: count the number of patients available in training/test sets, and there is your sample size. But that ignores the question of prevalence, i.e., the unbalanced datasets. In medicine, the sample sizes in two classes are typically unbalanced, sometimes dramatically. Consider, for example, the STRIVE study by company GRAIL (<https://grail.com/clinical-studies/>), which aims to recruit 120,000 breast cancer patients. This is gigantic by any standards.

But based on breast cancer incidence data, only about 600 (0.5%) women in the study will be diagnosed with cancer during the study duration. Therefore the effective sample size is closer to 1,200 (600 “positive” and 600 “negative”). Of course, you can use all the 100,000+ negatives, but the gain from that is only better estimate of specificity (at a cost of slower learning). To fully characterize the test, one needs sensitivity, which is driven by the number of “positive” patients, which is just around 600. In other words, the effective sample size is driven by the cardinality of the smallest class, not total.

Back to the second question: what is the threshold for Big Data? It is, of course, a judgment call, but my guess is that couple of hundred samples per class does not qualify as Big Data in terms of sample sizes. I’ll also say that the small sample sizes are a fundamental limitation due to time-to-outcome and epidemiology (prevalence of diseases), not technological obstacle.

Methods and Software

Approaching PM as Small Data problem has significant implications for machine learning methods and software. That could easily fill another blog, or even a book. In my view, a key implication is that the common split of data into training, validation and test is problematic in this context. The reason is that if you have say 100 samples per class, and split them, for example, 60/20/20, then the performance estimates obtained on the 20 validation/test samples are hugely variable and certainly not sufficiently robust to develop diagnostic products for use on patients. Another challenge: if a small validation set is used repeatedly for model tuning, it becomes, in effect, a training set, because the model becomes tuned to the validation set, not future populations of patients. With millions or thousands of samples per class available, this would be basically non-issue; but with low hundreds, or less, it becomes a major problem.

My approach is the use of elaborate cross-validation techniques, but that is certainly not the only way. I’ll just add that in general I have not found adequate support for small sample analyses in leading machine learning frameworks, and invariably had to develop in-house solutions for my work.

PAQ (Potentially Asked Questions)

This section is similar to FAQ, except that no one actually asked me these questions, so I can’t call them “Frequently Asked”. But I expect they might be asked.

Isn’t all this obvious? Doesn’t everyone understand the need to have outcome data?

You would think. Despite that, to my knowledge genomic PM has not been commonly characterized as Small Data in publications, talks, blogs etc. In particular I haven’t heard much discussion in public sphere around the key problem of time-to-outcome. Of course, plenty of my colleagues who work in the field are fully aware of these limitations, but I thought it’d be helpful to bring them up in an accessible blog format.

What About Other Applications, Like NIPT and Immunotherapy? Don’t they prove you can practice Precision Medicine without ML?

Yes, PM using genomic data can and is being practiced without ML. The NIPT (Non-invasive Prenatal Testing) is a major success story. But that’s an exception – otherwise, the record is mixed at best.

For example, in cancer care the approach is this: consider drugs which target particular mutations. Since we know the target, we only give the drugs to people with those genomic variations. Hence there is no need for any ML, and the Big Data/Small Data argument is moot.

In principle, this is very elegant idea because it is much simpler than ML. But so far, the record isn’t stellar (see for example https://jamanetwork.com/journals/jamaoncology/article-abstract/2713848 and excellent podcast https://itunes.apple.com/us/podcast/plenary-session/id1429998903?mt=2, Episode 1.20). I don’t find that surprising because it seems implausible that a treatment response of human being, composed of trillions of cells and countless interactions, would be accurately predicted by a single genomic feature.

Bottom line: single gene methods work for NIPT, but otherwise to a very limited extent. For truly precise targeting, it seems necessary to learn from data – at least until we master biology such that we can accurately predict outcomes from first principles. Not an easy trade-off, but that’s where we are.

What about imaging? Isn’t there plenty of phenotype data there?

Possibly. I focused on genomics in this blog, but couple of points regarding imaging: there is indeed plenty of annotated images available in public databases, various ML challenges and elsewhere. The crucial question is what is considered “outcome”. If an outcome is an expert-annotated feature of image (for example, “opacity” in X-ray images), then there is large amount of data, and it is available very soon after image is taken. In that case the key argument (that it takes long time to acquire phenotypic data for PM) does not apply. Hence, those applications may indeed take advantage of Big Data (and some have; we should see in the near future if they are adopted by physicians). However, if an outcome is a patient-centered feature, like onset of symptoms, or clinical event (stroke, cancer, confirmed diagnosis of Alzheimer’s etc.), then the phenotype takes a long time to observe, and the available numbers become dramatically smaller. Bottom line: I don’t know if there is a clear answer on this but we may know more relatively soon.

Isn’t there some health data available for people whose genomes are in public databanks? That’s a lot of people – sounds like Big Data

There is indeed some phenotype data for almost all samples in the large databases. But to develop a clinical PM test, you need exact outcomes for a particular condition, not generic health data. For example, for those working on cardiovascular diseases, the large collection of tumor genomes+diagnoses in TCGA (The Cancer Genome Atlas) isn’t too helpful, and vice versa. The numbers rapidly become small as soon as you focus on a specific disease.

What about EHRs? Isn’t that truly Big Data PM, with many patients?

Quite possible in the near/medium future, but at present, seems too early to tell: “Can you really tell how good a new medicine is from Flatiron’s data? Even Nat and Zach admit that the jury is still out.” (https://www.forbes.com/sites/matthewherper/2018/11/14/at-24-two-entrepreneurs-took-on-cancer-at-32-theyre-worth-hundreds-of-millions/#383c9a42d689)

Why are people collecting all these genomes? Isn’t that a waste of time, if there are no phenotypes?

Certainly not waste of time – crucial for research in biology, drug discovery, genome association studies etc. But most of the genomic Big Data, I am arguing, isn’t directly applicable for genome-driven patient care (i.e., PM) due to lack of usable outcomes.

How about unsupervised learning? It doesn’t need phenotypes

That is very far from clinical applications. Who knows what happens in far future, but for the time being, there is no unsupervised PM.

Machine Learning with a Twist: Detecting Structural Differences in Complex Data

Supervised learning, the most developed form of machine learning, starts with the preparation of labeled training data that are rich and representative (e.g., a large number of typical “spam” or “normal” emails). The machine teaches itself with these examples, such that it can label new data (i.e., new emails) that are unseen but similar to the training data. If the new data turned out to be different (say, spams with other patterns), typical advice would be to enrich the training set, such that it reflects the real process that generates the data we would like to label.

The above approach works that way because the ultimate goal there is to automate the labeling of new data, as accurately as possible. But, accurate automation on new data is not always the goal. What if, instead, we are interested in the discrepancy between the training data and new data?

In a new research paper, “Reading China: Predicting Policy Change with Machine Learning,” we show that machine learning, with a twist, can uncover structural changes underlying complex data. We train a neural network algorithm that “reads” the People’s Daily, the mouthpiece of the Communist Party of China, and classifies whether each article appears on the front page — part of the People’s Daily editor’s job. It turns out that such an algorithm can be used to detect changes in the newspaper’s issue priorities, which, in turn, have profound implications for China’s government policies.

The algorithm tries to mimic the mind of an avid People’s Daily reader who reads its articles and tries to figure out how its editor prioritizes them. If the reader had digested, say, five years’ worth of articles, they would be able to say, with a certain degree of confidence, what’s on the editor’s mind and what kind of articles do or do not “deserve” the front-page status.

But what we are really interested in is whether such an editorial paradigm has shifted over time. To answer this question, suppose the reader applies the paradigm acquired from the five years of observations (the training data) on the articles published in the next quarter (the new data). The reader may be surprised by the new articles; their educated guess about the front page may turn out exceptionally good or particularly poor — a signal of change. While a small surprise may well be taken as noise, a strong signal would convince the reader that their understanding of the editor’s mind is no longer valid and that the priorities of the People’s Daily have fundamentally changed.

We construct a quarterly indicator, which we call the Policy Change Index (PCI) of China, that captures the amount of surprise the algorithm has in each quarter, compared to the previous five years. The key to the design of the PCI is the realization that changes in underlying patterns come from the discrepancy — not the consistency — between what the algorithm has learned and what it will encounter.

The namesake of the indicator comes from the fact that detecting changes in the newspaper’s priorities allows us to predict changes in the Chinese government’s policies. This is because the People’s Daily is at the nerve center of China’s propaganda system, an essential function of which is to mobilize resources to attain the government’s policy goals. Moreover, before major policy changes are made, the government often finds it necessary to justify to or convince the public that those changes are the right moves for the country. Hence, while our algorithm is detecting propaganda change in real time, it’s really predicting policy changes for the future.

When put to the test against the ground truth — policy changes in China that did occur in the past — the PCI could have correctly predicted many of the most critical junctures in the history of China’s economy and reforms. These major changes include the beginning of the Great Leap Forward in 1958, that of the economic reform program in 1978, and, more recently, a reform speed-up in 1993 and a reform slow-down in 2005, among others.

Once we allow for the discrepancy between the training data and new data, the machine learning tools open the door to a variety of applications where detecting structural changes in complex data is the central question.

The interested reader can find more details about China’s policy changes, our methodology, and its potential applications in our research paper or the website of our project. We have also released the source code of the project on GitHub, so that the academic, business, and policy communities can not only replicate our findings but also apply our method in other contexts.


Julian TszKin Chan

Let’s Talk Bayesian Optimization



As a machine learning practitioner, “Bayesian optimization” has always been equivalent to “magical unicorn” that would transform my models into super-models. So off I went to understand the magic that is Bayesian optimization and, through the process, connect the dots between hyperparameters and performance.

Through hyperparameter optimization, a practitioner identifies free parameters in the model that can be tuned to achieve better model performance. There are a few commonly used methods: hand-tuning, grid search, random search, evolutionary algorithms and Bayesian optimization. Hand-tuning is a manual guess-test-revise process that relies on a practitioner’s previous experience and knowledge. A grid search exhaustively explores configurations until a reasonable accuracy has been reached. Random search uses random samplings of all possible combinations of all hyperparameters. And evolutionary algorithms use the comparison of “mutation-like” configurations with the best performing configurations to iterate on the model parameters. The goal of each of these methods is to find the global optima of hyperparameter values.

The problem, however, is that each of these methods has a fatal flaw. Hand-tuning results may not be reproducible from practitioner to practitioner, and the process is difficult to scale. Grid search cannot efficiently optimize models with more than 4 dimensions, due to the curse of dimensionality [5]. Although random search is more capable than grid search, its naive approach is still both time-consuming and expensive and is more likely to settle in a local optima. Furthermore, it is unscalable beyond 10 parameters [5]. Evolutionary algorithms are great if you have access to nearly unlimited compute that can be run in parallel, but is often difficult to implement if you do not.  

Bayesian optimization democratizes access to these algorithmic superpowers by relaxing each of these constraints. Originally popularized as a way to break free from the grid, Bayesian optimization efficiently uncovers the global maxima of a black-box function in the defined parameter space. In the context of hyperparameter optimization, this unknown function can be the objective function, accuracy value for a training or validation set, loss value for a training or validation set, entropy gained or lost, AUC for ROC curves, A/B test results, computation cost per epoch, model size, reward amount for reinforcement learning, among others. Here is a quick visual summary of how Bayesian optimization works:

Step 1:

Initialize process by sampling the hyperparameter space either randomly or low-discrepancy sequencing and getting these observations [3].

Step 2:

Build a probabilistic model (surrogate model) to approximate the true function based on given hyperparameter values and their associated output values (observations). In this case, fit a Gaussian process to the observed data from step 1. Use the mean from the Gaussian process as the function most likely to model the black box function.

Step 3:

Use the maximal location of the acquisition function to figure out where to sample next in the hyperparameter space. Acquisition functions play with the trade-off of exploiting a known maxima and exploring uncertain locations in the hyperparameter space. Different acquisition functions take different approaches to defining exploration and exploitation.

Step 4:

Get an observation of the black box function given the newly sampled hyperparameter points. Add observations to the set of observed data.

This process (Steps 2-4) repeats until a maximum number of iterations is met. By iterating through the method explicated above, Bayesian optimization effectively searches the hyperparameter space while homing in on the global optima. [6, 1, 2].

There are a variety of attributes of Bayesian optimization that distinguish it from other methods. In particular, Bayesian optimization is the only method that

  1. Adaptively and intelligently explores a hyperparameter space with optimal learning.
  2. Explore and exploit to find global optima [6, 2].
  3. Robustly handles noisy data.
  4. Optimally searches non-continuous and irregular spaces.
  5. Efficiently scales with hyperparameter domain.
  6. Maximizes utilization of computing resources.

 

How does this work in practice? Consider a CNN architecture for sentiment analysis [4]. This experiment tunes hyperparameters for each component of the modeling process from preprocessing (embedding dimension) to architecture (filter sizes and feature maps) to training (learning rate and dropout), and compares the results of random search, grid search, and SigOpt’s proprietary set of Bayesian optimization algorithms. The results are as follows:

Experiment Type Accuracy Trials Epochs CPU Time CPU Cost GPU Time GPU Cost
Default (No tuning) 75.70 1 50 2 hours $1.82 0 hours $0.04
Grid Search (SGD Only) 79.30 729 38394 64 days $1401.38 32 hours $27.58
Random Search (SGD Only) 79.94 2400 127092 214 days $4638.86 106 hours $91.29
Bayesian Optimization Search (SGD Only) 80.40 240 15803 27 days $576.81 13 hours $11.35
Grid Search (SGD + Architecture) Not Feasible 59049 3109914 5255 days $113511.86 107 days $2233.95
Random Search (SGD + Architecture) 80.12 4000 208729 353 days $7618.61 174 hours $149.94
Bayesian Optimization  Search (SGD + Architecture) 81.00 400 30060 51 days $1097.19 25 hours $21.59

From the above results, we see that tuning beats no-tuning, with a max of 5.3% increase in accuracy. It is also clear that Bayesian optimization beats grid and random search in accuracy, cost and computation time on both CPU and GPU. Bayesian optimization is able to achieve around a 1-2% boost in accuracy compared to grid and random search for 12%-14% the cost of random search on CPU and GPU. Furthermore, Bayesian optimization arrives at the global optima in a fraction of the time, allowing you to test out more models and architectures. This is only one data point, but we see this pattern of Bayesian optimization being more skillful and powerful replicated in scenarios such as regression models, reinforcement learning, unsupervised learning, and deep learning. Here is more information on common problems with hyperparameter optimization and how to evaluate hyperparameter optimization strategies.

Now that I’ve convinced you to use Bayesian optimization for your hyperparameter optimization, here are some tips to make Bayesian optimization easy.

  1. Choose the right metric or metrics to optimize
  2. Engineer around parameterization for easy tuning
  3. Consider tuning as an integral part of your ML workflow instead of a final step of your modeling

 

At SigOpt, we’ve created an easy to integrate and easy to use API that also supports advanced Bayesian optimization methods such as constraints, conditional parameters, multi-metric optimization, and parallelism. Happy modeling!

References

[1] E. Brochu, V.M. Cora, N. de Freitas. A tutorial on Bayesian optimization of expensive cost functions, with application to active user modeling and hierarchical reinforcement learning. CoRR, abs/1012.2599, 2010.
[2] P. Frazier. Bayesian Optimization. Recent Advances in Optimization and Modeling of Contemporary Problems, October 2018.
[3] M. W. Hoffman, B. Shahriari. Modular mechanisms for bayesian optimization. In NIPS Workshop on Bayesian Optimization, 2014.
[4] Y. Kim. Convolutional Neural Networks for Sentence Classification. In Proceedings of ACL 2014. June 2014.
[5] I. Dewancker, M. McCourt, S.Clark, P. Hayes, A. Johnson, G. Ke. A Stratified Analysis of Bayesian Optimization Methods. arXiv:1612.04451. December 2016.
[6] B. Shahriari, K. Swersky, Z. Wang, R. P. Adams, and N. de Freitas. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE, 104(1):148–175, Jan 2016.

Thank you to essential contributions from Harvey Cheng, Alexandra Johnson, Mike McCourt, Kevin Tee.

MLconf SF 2018 Speaker Resources

2018 Speaker Resources


Ilke Demir, Postdoctoral Researcher, Facebook

Open datasets and call for papers:

DeepGlobe Benchmark

Generative Street Addresses

My related references

  • Demir I., Hughes F., Raj A., Dhruv K., Muddala S.M., Garg S., Doo B., Raskar R., 2018. Generative Street Addresses from Satellite Imagery. ISPRS International Journal of Geo-Information (IJGI).
  • Aliaga D., Demir I, Benes B., Wand M., 2016. Inverse Procedural Modeling of 3D Models for Virtual Worlds. ACM SIGGRAPH 2016 Courses. (SIGGRAPH)
  • Demir I., Aliaga D., Benes B., 2016. Proceduralization for Editing 3D Architectural Models. International Conference on 3D Vision 2016 (3DV).
  • Demir I., Aliaga D., Benes B., 2015. Coupled Segmentation and Similarity Detection for Architectural Models. ACM Transactions on Graphics (ToG), also SIGGRAPH 2015.
  • Demir I., Aliaga D., Benes B., 2015. Procedural Editing of 3D Building Point Clouds. IEEE International Conference on Computer Vision 2015 (ICCV).
  • Demir I., Aliaga D., Benes B., 2014. Proceduralization of Buildings at City Scale. International Conference on 3D Vision 2014 (3DV).

Mentioned projects

Mentioned references

  • [1] Stefan Voigt, Fabio Giulio-Tonolo, Josh Lyons, Jan Kucera, Brenda Jones, Tobias Schneiderhan, Gabriel Platzeck, Kazuya Kaku, Manzul Kumar  Hazarika,  Lorant  Czaran, et al. Global trends in  satellite-based emergency mapping. Science, 353(6296):247–252, 2016.
  • [2] Timnit Gebru, Jonathan Krause, Yilun Wang, Duyun Chen, Jia Deng, Erez Lieberman Aiden, Li Fei-Fei. Demography with deep learning and street view. Proceedings of the National Academy of Sciences Dec 2017, 114 (50) 13108-13113
  • [3] Carlos A. Vanegas, Ignacio Garcia-Dorado, Daniel G. Aliaga, Bedrich Benes, and Paul Waddell. 2012. Inverse design of urban procedural models. ACM Trans. Graph. 31, 6, Article 168 (November 2012), 11 pages.

 


Joan Xiao, Lead Machine Learning Scientist, Figure Eight

 


Prasanth Anbalagan, Senior Software Engineer (Q&E Analysis) on the Artificial Intelligence Center of Excellence Team, Red Hat

 


Yi Li, Dr. Yi Li, Machine Learning Research Scientist, Baidu Silicon Valley AI Lab

 


Dr. Leslie Smith, Senior Research Scientist, US Naval Research Laboratory

  • Smith, Leslie N. “Cyclical learning rates for training neural networks.” In Applications of Computer Vision (WACV), 2017 IEEE Winter Conference on, pp. 464-472. IEEE, 2017.
  • Smith, Leslie N., and Nicholay Topin. “Super-Convergence: Very Fast Training of Residual Networks Using Large Learning Rates.” arXiv preprint arXiv:1708.07120 (2017).
  • Smith, Leslie N. “A disciplined approach to neural network hyper-parameters: Part 1–learning rate, batch size, momentum, and weight decay.” arXiv preprint arXiv:1803.09820 (2018).

 

My github page:

 

Large batch papers:

  • Goyal, Priya, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He. “Accurate, large minibatch SGD: training imagenet in 1 hour.” arXiv preprint arXiv:1706.02677 (2017).
  • You, Yang, Igor Gitman, and Boris Ginsburg. “Scaling sgd batch size to 32k for imagenet training.” arXiv preprint arXiv:1708.03888 (2017).
  • Akiba, Takuya, Shuji Suzuki, and Keisuke Fukuda. “Extremely large minibatch sgd: Training resnet-50 on imagenet in 15 minutes.” arXiv preprint arXiv:1711.04325 (2017).
  • Codreanu, Valeriu, Damian Podareanu, and Vikram Saletore. “Scale out for large minibatch SGD: Residual network training on ImageNet-1K with improved accuracy and reduced time to train.” arXiv preprint arXiv:1711.04291 (2017).
  • Smith, Samuel L., Pieter-Jan Kindermans, Chris Ying, and Quoc V. Le. “Don’t decay the learning rate, increase the batch size.” arXiv preprint arXiv:1711.00489 (2017).
  • Jia, Xianyan, Shutao Song, Wei He, Yangzihao Wang, Haidong Rong, Feihu Zhou, Liqiang Xie et al. “Highly Scalable Deep Learning Training System with Mixed-Precision: Training ImageNet in Four Minutes.” arXiv preprint arXiv:1807.11205 (2018).
  • Yao, Zhewei, Amir Gholami, Kurt Keutzer, and Michael Mahoney. “Large batch size training of neural networks with adversarial training and second-order information.” arXiv preprint arXiv:1810.01021 (2018).

Tweet for a chance to win FREE books!

We’d like to thank the participating publishers who have created custom discount codes for online order and are sending books for the Book Display at MLconf 2018. At the close of the event, we’ll host a book giveaway for the attendees who tweet the most interesting and unique tweets, using the @mlconf or #mlconf18. Book winners will be tweeted at by the @mlconf handle on 11/14/18.

Manning Publications

The 40% off discount code on the following books (for all attendees):  ctwmlconf18

 

**Email books@mlconf.com for a chance to win a free eBook  Deep Reinforcement Learning in Action by Alexander Zai and Brandon Brown! https://www.manning.com/books/exploring-data-with-python?utm_source=mlconf&utm_medium=social&utm_campaign=book_exploringdatawithpython&utm_content=conference_distribution

 

CRC Press Machine Learning Books:

 

The books can be purchased from the CRC website, using the discount code MLC20. The discount is valid until December 31st 2018.

 

Cambridge University Press:

A 20% discount on Cambridge books in Machine Learning and related areas is available using the discount code MLCONF18. Check out classics such as:

For a full list of books, check out:  www.cambridge.org/MLCONF18

 

Packt Publishing

Packt is offering a 60% discount on ML books for the MLconf audience! Here: https://www.packtpub.com/tech/Machine-Learning 

Additional Related Books On Display:

 

Don't miss a thing!



Join our ML Community