Deep Learning

Guest Post by Kanika Vatsyayant: Diving Into the Future of Deep Learning: Exploring the Novel Challenges

Though it is believed that tech enthusiasts are passing through the golden age of Artificial Intelligence, engineers and scientists still need to explore miles progressing on their journey to Deep Learning projects.

Either it is calibrating time-series models, processing Bayesian deep learning models, or defining metrics for uncertainty evaluation, the existing capabilities of machine learning are all overestimated. In other words, the present expectations associated with machine learning are more of mystical assumptions.

Especially when we talk about the commercial applications of machine learning techniques such as deep learning, the approach is still very young. It needs an extensive amount of well-prepared and organized data to attain the accuracy we look for. This means understanding and adopting the dynamics of machine learning require rich access to resources and time with provision for risks.

Above all, the existing deep learning algorithms are suitable for working on training records and process the information using millions of parameters of the neural networks. This setup could turn vague when applied to new data. Thus, there are so many challenges associated with the idea of Deep Learning that must be worked on to attain futuristic outcomes.

Let us quickly dive into understanding some of the most significant challenges associated with deep learning and how a practical approach to AI testing and machine learning best practices could help attain progressive results. But before we start with understanding all the novel challenges, it is necessary to understand all the future advancements associated with deep learning.

The Future Innovations Surrounding Deep Learning

Deep learning is one of the most significant tech advancements. Going by the definition, the technology works by classifying the information imitating the crude behavior of the human brain through layered neural networks.

The neural networks then process the given set of input units to be fed on raw data like pictures, sound, or text to map the output nodes. And some of the most futuristic innovations that are surrounded by Deep learning could be listed as:

Attaining General AI or Hybrid AI
Supervised learning
Deep Learning is likely to give an all-new face to the healthcare and medicine industry with the power of data
Also, more innovation with the help of deep learning could be seen in the field of Fraud detection, virtual assistants, visual recognition, and entertainment.

All in all, deep learning has gone too deep into our day-to-day routine, and examples of the same could be tracked from some of the most well-known applications that are leveraging speech recognition, NLP, and image processing to solve problems more accurately.

However, reaching out to the extremities of deep learning and goals associated with Hybrid AI need thousands of hours of research, data mining, experiments, programming, failures, and restructuring, and everything that translates into the novel challenges associated with Deep learning implementations.

The Novel Challenges Associated with Deep Learning

When we talk about futuristic and intelligent machines that have their own brains, Deep Learning takes the front seat in any such development process. Even the most advanced AI applications are driven using the best of deep learning technology.

There are some novel challenges that are required to be addressed for reaching the futuristic applications of Deep Learning.

Machine Learning As An All-new Alchemy

The initial days of machine learning had some shallow practices where decision tree algorithms were ruled to make predictions. For instance, “If something is orange, citric, and is made of pieces, there’s high likeliness for that to be an Orange.” Though these models didn’t have extraordinary intelligence to recognize differences between a Mandarin Orange or Cara Cara, it was easy to understand how they worked.

But Deep Learning is different as it works on hierarchical data representations. However, the only thing that restricts creating such technology is the lack of understanding of how it can be achieved.

The problem is defined as Black Box as AI supervisors have the knowledge of input and outputs while engineers know how a single prediction is nurtured, but no one understands how the entire model works. And this has turned to be a hurdle into the path of creating futuristic products like automated credit assessments, driverless vehicles, and medicine development.

Moreover, several researchers have claimed that AI is likely to encourage negative emotions and fear with people being scared of objects or technology behaving just like a human. Thus, it would be nothing wrong to say that this “Uncanny Valley” has enabled humans not to accept technology that can do typical human stuff.

Though the situation will change with the generations becoming more familiar with the digital environment surrounded by Machine learning, beating Black Box at present is a challenge that engineers and scientists are trying to overcome.

Resource Deficiency: Lack of Expertise

Though deep learning, has managed to capture the attention of business and tech enthusiasts across the globe, only a few specialists have managed to master the development process. Moreover, data scientists who have the proper knowledge of machine learning, most of them lack the knowledge of software engineering.

The situation has now turned worse as Element AI in a report shared by Bloomberg in 2018, estimated that only fewer than 10,000 experts have the skill to tackle AI research, and the most prominent industry players like Amazon, Facebook, Google, Microsoft had already reserved the best out of stock.

This means a massive jump in the salaries for AI experts, making machine learning and deep learning much difficult technologies to foster. New York Times reported that experts working on AI with only a few years of experience tended to earn up to $500,000 every year back in 2017, while the most experienced ones were likely to earn as high as NBA superstars.

Need For Extensive Training Data

Training Deep Learning models need access to extensive data sets. Though it may appear to be an easy task to accommodate or store information, purchasing the most relevant data can be expensive. Besides, it is equally consuming to prepare data for the training sets, especially when you need to work on clusters, regression, and classification.

The process even involves the need for consistent formatting and mechanism to collect information so that tasks like aggregation, attribute sampling, and record sampling could be done, which may need data decomposition and rescaling. All in all, the process needs to buy the time of the most skilled engineers, which might feel heavy on your investment graph.

In case the organization aims to use their stored data for the process, chances are high that privacy issues intervene, leading to legal issues with the people to whom the data belongs, even when they shared consent at the early stages. Thus, using personal data could lead to risks and unwanted expenses on data protection regulations like European General Data Protection Regulation.

Optimization Requirements Related To Deep Learning Model Performance

Though machine learning may appear to be a magic wand that tech giants like Alphabet and Microsoft are trying to aim with TensorFlow and ONNX, running these systems needs data from around the world. Moreover, all these efforts could not deny the fact that deep learning is very young, or rather we can say it is immature to be called production-ready.

Especially, when we talk about working on all the hyperparameters associated with the learning process, any changes made on values could lead to massive alterations in the model performance. In other words, the inability to work on hyperparameter optimization could significantly hamper the performance of deep learning models. Besides, the lack of knowledge and resources to detect the change requirements could affect the optimization tactics making way for performance degradation.

The AI Bias & Roll-Out Pressure

Since Machine Learning is all about utilizing the best of AI, the future of machine learning and AI systems entirely depends on the quality of data on which systems are trained. Moreover, the data which is used in the production process often involves the issue of bias as it comes from organizations that have filters on demographic specifications.

Ample Time & Planning Requirements

When we talk about software from the past, the development was quite easier as compared to modern software integration requirements where ML and AI are ruling the tech world.

Moreover, the bigger challenge of working on deep learning solutions is time requirements. On top of that, the uncertainties involved in the development process make it difficult to attain precision. Even the most refined engineers who have been working on machine learning for years tend to struggle with analyzing data sets and confirm that the training models could replicate results for different data.

Best Practices To Tackle Deep Learning Challenges In The Future

Work On Your Training Data

The primary objective of the machine learning engineers should be identifying the cases where data is misclassified by the algorithm. These are most likely to be edge cases and mislabels which need extensive testing on training data and models. The process of analyzing and processing training data with the algorithm usually aims to work on use cases where the model is not able to perform better than independent humans.

End-to-end Analysis & Deployment

The next most important factor that should be worked through Machine Learning includes working on the simplest areas of the entire deployment plan. This would help you overcome any model complexity as you progress in the documentation, as real-world modeling often needs you to work much harder in real scenarios. However, the end-to-end analysis allows you to work on every piece of complexity.

Besides, any deep learning models that are worked in regard to the end user’s perspective could help to identify crucial issues with the model. Also, the approach helps to reassess the training data collected so that any issues with the information could be worked.

Find Ways To Work Inevitable Cases Of Algorithm Fails

When we say Deep Learning, a good number of efforts usually fail, and therefore it becomes necessary to handle all these factors effectively. However, to build a confidence score for the test cases that work well, leaning on the batch process could be done. This would aid the collection of high-quality training data, accommodate less confident predictions, and highlight red flags.

Concluding It All…

Aligning with the Deep Learning technology for unleashing contemporary machine learning requires a thorough understanding of the limits. Achieving the technique to master the way machine learning feed on data could help turn some of the most futuristic technology changes. This could look like a future where self-driven vehicles are going to rule the roads or virtual assistants making way into our day-to-day operations at home or work.

Thus, taking over the novel challenges is vital and only needs an approach that could recognize how the AI industry, despite the extensive risk, has extensive rewards to offer. In a nutshell, technology enthusiasts only need to be patient, cautious, and respectful towards challenges that machine learning possesses while making way for the most predictive and precise solutions that have no empty promises involved.

Above all, there is a need for using existing machine learning technology to create deep learning of the future. The process could involve accelerating automation testing while encouraging the continuous quality culture, data-driven DevOps, and of course, autonomous end-to-end tests.

Deep Learning Infrastructure at Scale: An Overview

Many commercial applications of Deep Learning need to operate at large scale, typically in the form of serving deployed models to large numbers of customers. However inference is only half of the battle. The other side to this problem, which we will address here, is how to scale ML model building and research efforts when training datasets grow very large, and training must be done in a distributed fashion in the cloud. At Intuition Machines, we often need to deal with web-scale datasets of images and video, and as such having an efficient, scalable multi-user distributed training platform is essential.

This post encapsulates our recent review of the field, and is intended to serve as an introduction to the principles of training at scale along with a brief overview of the best available open source solutions for training your own networks in this manner.

WHAT ARE THE CHALLENGES WITH SCALING UP DEEP LEARNING?

As ML projects move from small scale research investigations to real world deployment, a large amount of infrastructure is required to support large scale inference, efficient distributed training, data ingest/transformation pipelines, versioning, reproducible experiments, analysis, and monitoring; creating and managing these support services and tools can eventually constitute much of the workload of ML engineers, researchers, and data scientists.

The portion of ML code in a real-world ML system is a lot smaller than the infrastructure needed for its support.

Source: Sculley et al.: “Hidden Technical Debt in Machine Learning Systems”

How can we enable these teams to focus on training the best models and delivering the best solutions without reinventing the wheel each time or getting bogged down in technical debt? (Sculley et al. discuss some other challenges in scaling ML systems, such as Glue Code, Pipeline Jungles, Reproducibility and Process Management Debt). We will take a look at some of the emerging infrastructural solutions to combat these issues below.

These days it is common to use cloud compute services such as Amazon AWS, Microsoft Azure, or Google GCP for handling all kinds of workloads. On the inference and data processing side, we can efficiently handle massive jobs in parallel using clusters of worker nodes and services co-ordinated with Kubernetes. We can be agnostic to underlying hardware and deploy on different platforms, using containerization to manage installation requirements.

For training, Deep Learning packages such as TensorFlow and PyTorch have added native support for multi-GPU training. This may be straightforward enough to deploy on a small scale on dedicated hardware, but when dealing with multi-node distributed training additional work is required. Another big requirement is portability: the same code should run on a laptop, a GPU rig, or a cluster, and even automatically scale based on demands. It is possible to hand-craft scripts to orchestrate deployment across a cluster, but this is cumbersome. However there are now open-source solutions that will enable packaging, containerization and automated deployment and co-ordination of workers and jobs. We will look at some in the next section.

Beyond distributed training of a single model, there are other requirements to make a truly efficient, repeatable, and flexible pipelined system for the whole ML workflow. When researchers are tackling new problems and investigating new models, it pays to have infrastructure in place that supports some of these tasks in order to streamline the process:

Data ingest, serving, versioning, preprocessing, splitting, transforms / augmentations
Job packaging and deployment; job reproducibility
Experiment monitoring/logging; comparison of results across runs, status dashboards
Automated hyperparameter tuning and model architecture search
Rapid deployment from training to online model serving / inference, and A/B testing
Support a wide variety of Deep Learning frameworks and other libraries in an integrated fashion, to avoid imposing limits on researchers enabling diverse solutions to be tried

Overall, the goal is to iterate and search over a variety of methods, models, features, and data pipelines in a manageable, repeatable, and measurable way, rather than trusting the more ad-hoc approaches that are common to many small research labs. Ideally after we have set up the processes for managing these components, the whole process becomes straightforward for researchers or data scientists to use the deployed environments in a self-service fashion, and to work collaboratively sharing a common resource pool of cluster nodes.

Some core architectural components of the ML workflow.

There are a large number of commercial solutions being touted to manage the ML lifecycle, such as RiseML, H20 Driverless, Anaconda Enterprise, Databricks, Paperspace, Domino Data Lab, ParallelM, neptune.ml, and comet.ml. Many of these platforms are also built around Kubernetes. There are also useful hosted platforms available on the major public cloud services, such as Microsoft’s Azure Machine Learning Service (formerly BatchAI), Amazon’s SageMaker, and Google’s Cloud AI, which help with some of these requirements.

Many large organizations have also developed their own private internal solutions to these problems, for example FBLearner Flow (Facebook), MichelAngelo (Uber), TFX (Google).

However, we are big fans of open source (having open sourced portions of our hCaptcha system and HUMAN protocol) and contribute to dozens of open source projects that we use internally. Therefore rather than using a hosted commercial service, or developing our own from scratch, our preference was to see what open source equivalent solutions are emerging in the ML infrastructure space. There are a variety of requirements here which may be handled by a single package, or a combination of several, which we will describe next.

SOME OPEN SOURCE ML INFRASTRUCTURE SOLUTIONS

JupyterHub

The Jupyter notebook paradigm has become popular among researchers and data scientists for rapidly iterating (especially Python) code, visualizing results, and experimenting with different models. JupyterHub allows hosting multiple users with their own notebook servers; it is designed to be flexible, scalable and portable, and can be deployed with Kubernetes.

KubeFlow

A project that originated at Google, Kubeflow began with the aim of making it easy to run TensorFlow jobs on Kubernetes, and is expanding to support a whole stack of other ML and infrastructure tools. These include:

TFJob – a resource/YAML file spec describing how to run a (possibly distributed) TensorFlow job on Kubernetes
Argo – Build container-native workflows on Kubernetes (every step of the workflow is a container). For example, build reusable containers automatically from trained models.
Seldon-core – model deployment via Docker; convert to microservices with REST/gRPC APIs; A/B testing; and manage deployed inference scalably.
TF Serving – another library serving models for inference, A/B testing, model versioning
Istio – metrics, A/B testing and integration with TF serving
Katib – Black Box Hyperparameter tuning, in the vein of Google Vizer
Ksonnet is used as an alternative to Helm for Kubernetes package management.

At the Deep Learning framework level, Kubeflow has been adding support for MXNet, PyTorch and Chainer as well as TensorFlow. Jupyterhub is also available as a component.

There are clearly a large number of separate components which are really standalone projects, with some overlaps and some gaps. The Kubeflow project, while more or less usable at the time of writing, is still under heavy development and requires a large investment of engineering time into understanding the system, configuring it for your needs, and often patching a few areas you need but no one has touched recently to make the most of everything in a truly integrated fashion, and to support multiple users dynamically.

Our view is that there is a lot of vision here and hopefully the pieces will come together rapidly as the project evolves. However, for larger organizations with distributed systems expertise it brings less benefit than you might expect due to the immaturity of the codebase and heavy churn. Without having a Kubeflow maintainer on staff, using it for non-trivial work may be difficult to justify and will certainly require a substantial time investment.

Polyaxon

Another integrated platform built on top of Kubernetes, that has an emphasis on reproducible ML at scale, in a simple and accessible way. It is somewhat more aligned to the goals of a self-service multi-user system, taking care of scheduling and managing jobs in order to make best use of available cluster resources. It also handles code/model versioning, automatic creation and deployment of docker images, and can support auto-scaling.

It provides features such as distributed model training, experiments, dashboards with visualization, metrics, and easy hyperparameter optimization. Polyaxon supports various DL frameworks including Tensorflow, Keras, MXNet, Caffe, PyTorch. Currently there is no built-in support for model serving, with more of an emphasis on experimentation and inference, but new features are being added rapidly.

Compared to Kubeflow, Polyaxon is very focused. Polyaxon doesn’t attempt to cover realtime serving, istio integration, has no ksonnet requirement, and isn’t tightly tied to seldon. Rather polyaxon is more focused on running adhoc or repeatable experiments. It does handle all of the package management and can automate notebook and tensorboard installs. Polyaxon solves the problem of running disparate jobs on the same cluster and it supports distributed training as well. It has a built in job management dashboard, as well as simple user management.

Polyaxon sits on top of Kubernetes services to provide an easy-to-use, repeatable ML experimentation and training platform, supporting dashboards, versioning and much more.

Source: https://docs.polyaxon.com/experimentation/concepts/

FfDL

IBM’s Fabric for Deep Learning (pronounced “fiddle”) is a Kubernetes based platform designed around elastic, cloud based distributed training, and abstracting some of the complexities of the underlying infrastructure to move towards “Deep Learning as a service”. It supports many of the main DL frameworks – Tensorflow, Keras, Caffe 1&2, and Torch, and makes use of Jupyter, Docker and Helm, and is designed around S3 based data and model storage. It provides a Grafana dashboard for monitoring jobs, and can use Horovod for distributed training. Manifest files are used to describe a model to be trained, hyperparameters, resource requirements, metrics, etc. REST APIs are used to interface with a gRPC Trainer, which stores jobs in a database, and handles scheduling multi-user training jobs across their life cycles, as well as parameter servers. There is a training data service to manage metrics and logs. For model deployment, integration with Seldon is supported.

Architecture of FfDL

Source: https://github.com/IBM/FfDL

Lore

This python framework was recently open sourced by Instacart, and is more of an easy to use workflow solution for the ML training and inference pipeline. It supports hyper-parameter search, data extraction, encoding and splitting pipelines, database connections, dependency management, CI/CD testing, model deployment, and integration with packages such as Keras, Tensorboard, Scikit-learn and Jupyter. One key idea is to standardize ML practice across different libraries, with a set of wrappers. There is perhaps more of an emphasis towards time series/textual data than video and images and CNN model based Deep Learning, and it doesn’t get into the world of containerization and cluster management.

As such, it is perhaps a good entry point for newcomers to the field who look for some of the consistency and workflow solutions we have been describing, but it perhaps lacks some of the more advanced features that are useful for an organization doing large scale distributed training and architecture search on visual domain data.

ML Flow

An open source library created by Databricks, designed for managing 3 stages of the ML lifecycle: Tracking, Projects, and Models. The goals of the project are to improve reproducibility and enable experiment tracking, and make it easier to move models to production. It is also designed to support any Python or R based ML framework, has a web UI to make tracking straightforward, and has multi-user support by running a tracking server. Tracking supports metrics, visualizations, storing artifacts, and comparing runs, as well as hyperparameter tuning. Projects support a simple way of packaging code, run parameters, and dependencies via YAML configuration files and Conda environments, and integration with Git. Projects also support scalable dataset I/O with hosts like AWS S3. Models can easily be deployed in several ways, including as a REST server, or exported into a Docker container,

Overall, it is easy to get started with a single-user local setup, but still has scalability to support larger datasets and teams. Distributed training or cluster support is not part of the functionality however, so additional work will be required here.

StudioML

This Python package from Sentient is designed to manage ML model building experiments, covering some of the same ideas as ML Flow or Lore, without deployment, but with some cloud-based training integration. It supports the most common packages – Keras, TF, PyTorch, and scikit-learn. Its main features include environment and dependency management, monitoring/logging, experiment management, hyperparameter search, artifact management, and containerization. One of its unique features is it has direct integration with Amazon EC2 and Google Cloud APIs, as well as auto-scaling to make launching its training jobs on these platforms straightforward.

Bighead

Airbnb recently presented their new open-source end-to-end ML pipeline library. As of the time of writing it has not yet been released, but looks very promising. It covers features across the whole workflow from data ingest/transformations, training, model management, hyperparameter tuning, monitoring and deployment. It is designed to be modular and support common ML training frameworks. There is support for built-in data transformations and data visualization tools, to help with one of the parts of the ML development process where researchers and engineers spend most of their time. Its Redspot environment is an extension of JupyterHub that helps with instance and package management, and built around automated Docker container services, which helps with the training workflow and multi-user environments. Model management keeps model code and trained models in sync, and improves reproducibility through common metrics. Production supports scalable streaming and brings consistency between training and deployment environments.

There is a lot of promise here, though we will have to see how everything behaves in practice once released.

Data

On the data ingest, processing, management, and retrieval fronts, there is a lot more than can be covered in detail here, in terms of how to effectively deal with training datasets once their scale outgrows what can be stored on a single node. For now we are making use of custom solutions in these areas, but there are a couple of packages worth pointing out.

Pachyderm

The ability to transform data using different pipelines is an essential part of the ML process. In experimentation, data engineering is often as if not more important than model design. As models and data evolve in parallel, many organizations face the problem of repeatability and versioning of datasets along with models. But it’s not possible just to store huge datasets in git. So packages such as Pachyderm enable data provenance, i.e. where did the data come from, and how has it been modified along the way. As data is ingested from different locations, cleaned, augmented, and transformed with different methods, version control can be applied, so that experiments can reveal if it was changes to the dataset or to a model that caused an improvement. Some of this may be seen as overkill in certain workflows, but with data being a key part of the whole ML process, it’s worth thinking how to integrate these approaches with experiment tracking, for instance. Pachyderm is also built around Kubernetes and can be integrated with Kubeflow.

Pachyderm allows data versioning and persisted model workflows.

source http://docs.pachyderm.io/en/latest/cookbook/ml.html

Solt

A library for image data augmentation. Often necessary in Computer Vision tasks, the ability to apply various parametric transformations to an image dataset can be a bottleneck in training. Being able to do this in a fast and streamable manner is a useful component.

OpenFaas

In keeping with the trend of using Kubernetes and Docker based microservices, the idea of Functions as a Service is a scalable way to create customized data processing pipelines from existing functional components, in a Serverless fashion. We have been investigating these approaches and integrating with other systems; this appears to be a flexible and robust way to handle large scale data processing requirements for both training and inference.

Distributed training

The solutions described so far help with managing training workflow, but taking advantage of additional compute power that a cluster can provide to give faster training results requires some changes to training code in order to transfer weight updates in stochastic gradient descent between workers. There are a couple of solutions that avoid having to hand-roll some of this process.

Distributed Tensorflow has capabilities for using a parameter server to manage distributed jobs. This requires specification of a ClusterSpec, containing IP addresses/ports of servers where the tasks should run. Projects like Kubeflow (via TFJob files) or Polyaxon can help set this up, on top of a Kubernetes cluster. This still requires a degree of interaction with the cluster setup itself. Some additional scripting is required for example to automate packaging.

Horovod

While the approach used by TensorFlow is typically to use a parameter server in order to manage

weight updates in a distributed training scenario, Uber’s Horovod library takes things to another level of performance and ease of use. It uses a different manner for averaging weights’ gradients across workers. Rather than all weights being averaged through a central parameter server, workers exchange only a portion of their gradient updates with their neighbors using a ring-allreduce algorithm, that makes much more efficient use of network bandwidth. It uses a Message Passing Interface like Open MPI, as well as using NVIDIA’s NCCL library to optimize these transfers both on a single Multi-GPU machine, as well as on a cluster.

Horovod uses the ring-allreduce algorithm to enable efficient distributed training without the need for a centralized parameter server

Source: https://eng.uber.com/horovod/

In terms of ease of use, there are only a few commands to inject into standard training code in order to support training in this way – standard optimizers are wrapped in a custom distributed version. Support was available for Tensorflow and Keras originally but now PyTorch sessions can be distributed in the same way.

Horovod has already proved itself in providing an effective enabling technology, and we would be excited to see it be natively supported by more of the ML lifecycle solutions.

Our use

At Intuition Machines, we are not strangers to the challenges of Machine Learning (ML) at scale. We have become experts at providing very large scale, efficient inference and data retrieval solutions to our clients, with a focus on Computer Vision, Images and Video.

For these tasks, we currently bootstrap our Kubernetes in Azure, and we use a combination of GPU nodes and virtual nodes to scale our laboratory cluster. We are integrating Polyaxon, with basic single sign on to manage our experiments, having easy access to the cluster. This provides a simple mechanism for users to start with a Jupyter notebook, and quickly transition it to distributed, large scale jobs, complete with experiment tracking and hyperparameter searches.

On the data side, we considered Pachyderm, which is more of a Hadoop rebuilt from scratch, and comes with some of the same downsides. Although Pachyderm is focused on “the data” it has its own mechanisms for processing this data. We currently already have a preprocessing pipeline, and we also use OpenFaas and an in-house system called Mongoose to process our large datasets resident on S3; integrating Pachyderm’s processing system so far didn’t buy us a lot.

Summary

Many of these packages, libraries and solutions have appeared during the last year, and most are still in Alpha or Beta stages. We expect to see them develop rapidly over the coming months. However, many are already in a state where they are useful for improving the real world workflows that many ML teams are facing. Larger organizations with more sophisticated requirements (especially on the data ingest side) will most likely benefit most from picking individual pieces to integrate into their preferred workflows rather than adopting opinionated and early-stage bundled solutions like Kubeflow.

Broadly speaking, we see that some of these solutions are designed with cloud-based training in mind while others are focused more on making the end-to-end ML workflow simpler for a single user system or a small research team. We are particularly excited to use solutions based around Kubernetes for the ability to make dynamic use of cloud resources, and efficiently deploy distributed training on a large scale.

However, smaller teams without in-house DevOps talent may be best served by choosing one of the simpler hosted solutions like Azure Machine Learning Service or Google Cloud ML Engine at the moment. Larger shops with more complicated needs are likely to find that the complete open source systems available are still very much in progress, and so picking and choosing components that add value on their own to plug into existing internal infrastructure will often be the most practical route for the near future.

Interview with Abhineet Gupta

Q1: Tell us briefly about yourself and your work.

AG: I am the Resilience Engineering Lead at One Concern, a Palo Alto-based benevolent AI company, with a mission to save lives and livelihoods before, during and after disasters. I have been involved with One Concern for almost 3 years, essentially since the company’s inception. I developed the resiliency framework that lays out the conceptual architecture for us to meet our vision to achieve planetary-scale resilience, quantify it at a community level, and constantly improve resilience over time. As part of this effort, I have created specific metrics for measuring how access to critical resources is impacted in disasters and to identify those communities that are most at risk before, during and after disasters. Besides working on resilience, I have also contributed to development of our Seismic product to estimate near real time impacts in an earthquake, created evaluation metrics for our machine learning models and set up the simulation pipeline to generate earthquakes for preparedness trainings and mitigation policy design.

Prior to One Concern, I completed my PhD thesis from Stanford on creating the framework to estimate community level seismic risk from man-made or induced earthquakes, with a case study implementation in Oklahoma. As part of this framework, I implemented and developed novel statistical methods to quantify how seismic rates change over time with changes in oil and gas extraction processes responsible for causing these earthquakes. Prior to my PhD, I worked as a structural design engineer at Magnusson Klemencic Associates in Seattle and was responsible for the design of an iconic building in Boise – Jack’s Urban Meeting Place – along with helping out with design of buildings in San Francisco, Delhi and Kuala Lumpur.

Q2: I had not previously heard of “resilience engineering.” Tell us a little bit about that field. Is it new? How many people worldwide graduate with a degree in resilience engineering?

AG: As far as I am aware, resilience engineering is not a degree offered by universities. Through our work at One Concern, we realized that most of the work around disaster impacts occurs in silos by hazard or by field of study, while resilience is an inherent characteristic of communities that spans across both expected disasters and unexpected events and across many fields like risk engineering, economics, data science, behavioral analysis, policy and education, urban design and social sciences. As a result, we decided to create this position and team at One Concern to focus on holistic resilience analysis. At One Concern, we have defined Resilience as the ability of communities to maintain their normal functionality and to adapt and grow in the face of acute shocks and long term stresses. We are also starting to see universities like Stanford, Harvard and MIT create multi-disciplinary programs to bring together experts from different fields and companies to work on resilience.

Q3: What is the story behind the founding of One Concern?

AG: One Concern spun off from a machine learning project in Andrew Ng’s class at Stanford. The co-founders, Ahmad Wani, Nicole Hu and Timothy Frank along with Yawar Aziz demonstrated in that project that rapid damage assessment after earthquakes was possible using available data and machine learning. Ahmad was also motivated to help people during disasters from his recent experience being stuck on his roof while floods in Kashmir inundated his town. Combined with the success of their project and encouragement from Andrew Ng, One Concern was founded to rapidly measure impacts after disasters, prepare for and mitigate against them and thus save lives and livelihoods.

Q4: What sort of open data do you use for your models? And what type of data do you (One Concern) collect on your own?

AG: We use a combination of open data, proprietary data and our self-collected data. Our open data sources include the US census data that we use for understanding demographics and building inventory, weather forecast data to estimate flood inundation and USGS earthquake data to simulate earthquakes.

One Concern has also created a Digital Anthropology team which is tasked with collecting disaster relevant data from across the world. Some of the regions where our team has been deployed include Bogota to collect data about buildings, demographics and socio-cultural parameters, and Mexico City and Lombok to collect post-earthquake damage data. We use a variety of data collection tools like apps, cameras and remote sensing imagery. A critical component of data processing for us is that individual sources do not accurately capture the required features of our models. As a result, our data processing team uses several statistical and data mining techniques to augment data from different sources to ensure that the final datasets represent the ground truth.

About Abhineet Gupta

Abhineet Gupta is the Resilience Engineering Lead at One Concern. He works on quantifying resilience through cities’ abilities to withstand stresses and shocks from natural and manmade disasters, and to provide the technological solutions to increase resilience. Abhineet graduated from Stanford in 2017 with a Master’s and PhD in earthquake engineering and PhD minor in Computer Science with a focus on artificial intelligence. For his PhD, he developed tools to evaluate seismic risk from man-made earthquakes in regions like Oklahoma. Before his PhD, he worked as a structural design engineer at Magnusson Klemencic Associates in Seattle and designed buildings in Boise, India and Malaysia.

LinkedIn
One Concern

Guest Blog by Alexander Tsyplikhin, Chief Operating Monster, DataMonsters

Deep Learning with Python by François Chollet

I occasionally give talks on deep learning, and I know how important it is to present the material in the right way — practical and easy to grasp. I was recently impressed by a new book on deep learning, and I decided to share it here.

The book is called “Deep Learning with Python”. The title would not catch my attention, if not the fact that the book is written by François Chollet. François Chollet is the author of Keras. Keras is an open-source deep learning framework for Python. Because of it’s beautiful simplicity, it is extremely popular: over two hundred thousand users, which is more than one third of all the deep learning engineers in the world.

“Deep Learning with Python” is a gem. I envy those who are just about to read it. It takes you through the journey of deep learning in a very captivating way. Here are the two things that I think make the book stand out compared to other deep learning books:

1. Easy. The material is targeted to anyone with basic Python coding experience. Mathematical concepts are explained using code snippets. No need to dive into math notation or equations, which can be inscrutable for people without a relevant degree.

2. Practical and advanced. Typically, advanced theoretical books are disconnected from concrete problems, and practical books are superficial.

In a solid way, the book covers the topics of:

Fundamentals of machine learning and neural networks,
Setting up your environment for practical work — both software and hardware,
Deep learning for computer vision — convolution operation is nicely explained,
Deep learning for text and sequences — both RNNs and CNNs, with advice on algorithmic trading (“don’t”),
Advanced deep-learning best practices — ways to be more in control,
Generative deep learning — both text and images (including “the smile vector”).

If you are interested in starting strong with deep learning, you will enjoy this book: https://www.manning.com/books/deep-learning-with-python

Read the original review here.

Alexander Tsyplikhin, PhD, has been working in the area of data science for fifteen years, building consumer and enterprise products. Alexander is currently the Chief Operating Monster of Data Monsters, a machine intelligence and data engineering consulting and implementation company. Data Monsters is a Preferred Deep Learning partner of NVIDIA, honoured to work with Cisco, Intel, Cray and other outstanding companies (see datamonsters.com).

Alexander holds a PhD in Speech Biometrics from MIPT and is passionate about using data to build great products.

Guest Blog by Keshav Dhandhania, Co-Founder, Compose Labs

Generative Adversarial Networks

In this article, I’ll talk about Generative Adversarial Networks, or GANs for short. GANs are one of the very few machine learning techniques which has given good performance for generative tasks, or more broadly unsupervised learning. In particular, they have given splendid performance for a variety of image generation related tasks. Yann LeCun, one of the forefathers of deep learning, has called them “the best idea in machine learning in the last 10 years”. Most importantly, the core conceptual ideas associated with a GAN are quite simple to understand (and in-fact, you should have a good idea about them by the time you finish reading this article).
In this article, we’ll explain GANs by applying them to the task of generating images. The following is the outline of this article

A brief review of Deep Learning
The image generation problem
Key issue in generative tasks
Generative Adversarial Networks
Challenges
Further reading
Conclusion

A brief review of Deep Learning

Sketch of a (feed-forward) neural network, with input layer in brown, hidden layers in yellow, and output layer in red.
Let’s begin with a brief overview of deep learning. Above, we have a sketch of a neural network. The neural network is made of up neurons, which are connected to each other using edges. The neurons are organized into layers – we have the hidden layers in the middle, and the input and output layers on the left and right respectively. Each of the edges is weighted, and each neuron performs a weighted sum of values from neurons connected to it by incoming edges, and thereafter applies a nonlinear activation such as sigmoid or ReLU. For example, neurons in the first hidden layer, calculate a weighted sum of neurons in the input layer, and then apply the ReLU function. The activation function introduces a nonlinearity which allows the neural network to model complex phenomena (multiple linear layers would be equivalent to a single linear layer).
Given a particular input, we sequentially compute the values outputted by each of the neurons (also called the neurons’ activity). We compute the values layer by layer, going from left to right, using already computed values from the previous layers. This gives us the values for the output layer. Then we define a cost, based on the values in the output layer and the desired output (target value). For example, a possible cost function is the mean-squared error cost function.

where, x is the input, h(x) is the output and y is the target. The sum is over the various data points in our dataset.
At each step, our goal is to nudge each of the edge weights by the right amount so as to reduce the cost function as much as possible. We calculate a gradient, which tells us how much to nudge each weight. Once we compute the cost, we compute the gradients using the backpropagation algorithm. The main result of the backpropagation algorithm is that we can exploit the chain rule of differentiation to calculate the gradients of a layer given the gradients of the weights in layer above it. Hence, we calculate these gradients backwards, i.e. from the output layer to the input layer. Then, we update each of the weights by an amount proportional to the respective gradients (i.e. gradient descent).
If you would like to read about neural networks and the back-propagation algorithm in more detail, I recommend reading this article by Nikhil Buduma on Deep Learning in a Nutshell.

The image generation problem

In the image generation problem, we want the machine learning model to generate images. For training, we are given a dataset of images (say 1,000,000 images downloaded from the web). During testing, the model should generate images that look like they belong to the training dataset, but are not actually in the training dataset. That is, we want to generate novel images (in contrast to simply memorizing), but we still want it to capture patterns in the training dataset so that new images feel like they look similar to those in the training dataset.

Image generation problem: There is no input, and the desired output is an image.
One thing to note: there is no input in this problem during the testing or prediction phase. Everytime we ‘run the model’, we want it to generate (output) a new image. This can be achieved by saying that the input is going to be sampled randomly from a distribution that is easy to sample from (say the uniform distribution or Gaussian distribution).

Key issue in generative tasks

The crucial issue in a generative task is – what is a good cost function? Let’s say you have two images that are outputted by a machine learning model. How do we decide which one is better, and by how much?
The most common solution to this question in previous approaches has been, distance between the output and its closest neighbor in the training dataset, where the distance is calculated using some predefined distance metric. For example, in the language translation task, we usually have one source sentence, and a small set of (about 5) target sentences, i.e. translations provided by different human translators. When a model generates a translation, we compare the translation to each of the provided targets, and assign it the score based on the target it is closest to (in particular, we use the BLEU score, which is a distance metric based on how many n-grams match between the two sentences). That kind of works for single sentence translations, but the same approach leads to a significant deterioration in the quality of the cost function when the target is a larger piece of text. For example, our task could be to generate a paragraph length summary of a given article. This deterioration stems from the inability of the small number of samples to represent the wide range of variation observed in all possible correct answers.

Generative Adversarial Networks

GANs answer to the above question is, use another neural network! This scorer neural network (called the discriminator) will score how realistic the image outputted by the generator neural network is. These two neural networks have opposing objectives (hence, the word adversarial). The generator network’s objective is to generate fake images that look real, the discriminator network’s objective is to tell apart fake images from real ones.
This puts generative tasks in a setting similar to the 2-player games in reinforcement learning (such as those of chess, Atari games or Go) where we have a machine learning model improving continuously by playing against itself, starting from scratch. The difference here is that often in games like chess or Go, the roles of the two players are symmetric (although not always). For GAN setting, the objectives and roles of the two networks are different, one generates fake samples, the other distinguishes real ones from fake ones.

Sketch of Generative Adversarial Network, with the generator network labelled as G and the discriminator network labelled as D
Above, we have a diagram of a Generative Adversarial Network. The generator network G and discriminator network D are playing a 2-player minimax game. First, to better understand the setup, notice that D’s inputs can be sampled from the training data or the output generated by G: Half the time from one and half the time from the other. To generate samples from G, we sample the latent vector from the Gaussian distribution and then pass it through G. If we are generating a 200 x 200 grayscale image, then G’s output is a 200 x 200 matrix. The objective function is given by the following function, which is essentially the standard log-likelihood for the predictions made by D:

The generator network G is minimizing the objective, i.e. reducing the log-likelihood, or trying to confuse D. It wants D to identify the the inputs it receives from G as correct whenever samples are drawn from its output. The discriminator network D is maximizing the objective, i.e. increasing the log-likelihood, or trying to distinguish generated samples from real samples. In other words, if G does a good job of confusing D, then it will minimize the objective by increasing D(G(z))in the second term. If D does its job well, then in cases when samples are chosen from the training data, they add to the objective function via the first term (because D(x) would be larger) and decrease it via the second term (because D(x)would be small)
Training proceeds as usual, using random initialization and backpropagation, with the addition that we alternately update the discriminator and the generator and keep the other one fixed. The following is a description of the end-to-end workflow for applying GANs to a particular problem

Decide on the GAN architecture: What is architecture of G? What is the architecture of D?
Train: Alternately update D and G for a fixed number of updates
1. Update D (freeze G): Half the samples are real, and half are fake.
2. Update G (freeze D): All samples are generated (note that even though D is frozen, the gradients flow through D)
Manually inspect some fake samples. If quality is high enough (or if quality is not improving), then stop. Else repeat step 2.

When both G and D are feed-forward neural networks, the results we get are as follows (trained on MNIST dataset).

Results from Goodfellow et. al. Rightmost column (in yellow boxes) are the closest images from the training dataset to the image on its direct left. All other images are generated samples.
Using a more sophisticated architecture for G and D with strided convolutional, adam optimizer instead of stochastic gradient descent, and a number of other improvements in architecture, hyperparameters and optimizers (see paper for details), we get the following results

Results from Alec Radford et. al. Images are of ‘bedrooms’.

Challenges

The most critical challenge in training GANs is related to the possibility of non-convergence. Sometimes this problem is also called mode collapse. To explain this problem simply, lets consider an example. Suppose the task is to generate images of digits such as those in the MNIST dataset. One possible issue that can arise (and does arise in practice) is that G might start producing images of the digit 6 and no other digit. Once D adapts to G’s current behavior, in-order to maximize classification accuracy, it will start classifying all digit 6’s as fake, and all other digits as real (assuming it can’t tell apart fake 6’s from real 6’s). Then, G adapts to D’s current behavior and starts generating only digit 8 and no other digit. Then D adapts, and starts classifying all 8’s as fake and everything else as real. Then G moves onto 3’s, and so on. Basically, G only produces images that are similar to a (very) small subset of the training data and once D starts discriminating that subset from the rest, G switches to some other subset. They are simply oscillating. Although this problem is not completely resolved, there are some solutions to it. We won’t discuss them in detail here, but one of them involves minibatch features and / or backpropagating through many updates of D. To learn more about this, check out the suggested readings in the next section.

Conclusion

I hope that in this article, you have understood a new technique in deep learning called Generative Adversarial Networks. They are one of the few successful techniques in unsupervised machine learning, and are quickly revolutionizing our ability to perform generative tasks. Over the last few years, we’ve come across some very impressive results. There is a lot of active research in the field to apply GANs for language tasks, to improve their stability and ease of training, and so on. They are already being applied in industry for a variety of applications ranging from interactive image editing, 3D shape estimation, drug discovery, semi-supervised learning to robotics. I hope this is just the beginning of your journey into adversarial machine learning.

Keshav is a cofounder of Compose Labs (commonlounge.com) and has spoken on GANs at international conferences including DataSciCon.Tech, Atlanta and DataHack Summit, Bangaluru, India. He did his masters in Artificial Intelligence from MIT, and his research focused on natural language processing, and before that, computer vision and recommendation systems.

Guest Blog by Alan Descoins, CTO, Tryolabs

Why We Built an Open Source Computer Vision Toolkit

Introduction

Over the last few years, the Machine Learning (ML) landscape has changed dramatically. The comeback of neural networks in the form of Deep Learning has opened new creative ways to approach many classical ML and Artificial Intelligence (AI) problems.
Out of all the areas vastly improved by Deep Learning techniques, computer vision has been particularly revolutionized by major breakthroughs that have only very recently occurred.

The Deep Learning revolution in computer vision

It all started when a new model published in a paper by Krizhevsky et al. — known as AlexNet — won the ImageNet challenge in 2012. This challenge had been running for quite some number of years, and one parts consists of an image classification task. In it, algorithms are given images and must classify them into a fixed set of categories. The training set consists of 1.2 million images, each of which can be labeled into 1000 categories. For the evaluation, the algorithm must decide the categories of images which it hasn’t previously seen during training.
This new AlexNet model could reduce the error rate from 25% in 2011 to a mere 16%, greatly improving the results of prior state of the art techniques which, at the time, still relied on handcrafted features such as HOG or SIFT. Subsequent years continued the trend, and we quickly saw the error rate drop to less than 3% — often surpassing human performance in the narrow task of assigning a category to images. Deep Learning was here to stay, image classification was “solved” and this xkcd comic was obsolete.
What changed? It is interesting to note that conceptually, AlexNet was almost equivalent to a model called LeNet-5, proposed by Yann LeCun (now Director of AI Research at Facebook) almost 15 years earlier, used back then to recognise handwritten digits. However, LeNet had roughly 2 orders of magnitude less parameters (AlexNet had around 60 million), and it had a much smaller dataset on which to train (MNIST). Advances in modern GPU technology, added to the availability of larger open datasets (such as ImageNet) have really driven algorithmic advancements which ended up significantly improving accuracy in almost every area where Deep Learning has been applied.
In particular, both LeNet and AlexNet belong to a family of neural network architectures called Convolutional Neural Networks (CNNs or ConvNets). These networks have a number of properties that make them really well suited for image processing tasks. CNNs use a layered structure, in which each layer has the ability to slide different filters through the output of the previous layer. This allows the networks to learn patterns of increasing level of abstraction, until they can grasp complex concepts that would otherwise be hard to express. For example, it has been shown that lower layers learn simple features like edges and color gradients, while higher layers can learn more abstract features like eyes or faces.

On object detection in images

The power of CNNs has also been applied to object detection, dramatically improving results over previous methods. In object detection, we aim to identify a variable number of objects present in an image, and for each one, classifying it into a fixed set of categories and also drawing a bounding box that encloses it. This is a much harder problem than image classification, and so far it has not been “solved”.

This problem has plenty of applications, ranging from simple counting any type of object (like people or cars), aerial image analysis, visual search engines or health care, where it can help with analysis of medical images. Did we also mention self driving cars?
One can arguably say that with the recent explosion in Deep Learning frameworks (Google’s TensorFlow, Facebook’s PyTorch or Caffe2, Microsoft’s CNTK, Amazon’s MXNet, among many others) and the tools built around them it is fairly straightforward for most programmers to implement state of the art image classification techniques into applications. However, object detection is a tougher nut to crack.

Object detection in the industry

Over the last couple of years, we have identified some issues most companies face when incorporating these technologies into their platforms.
Intense learning curve
Even for people with tons of experience (and even with PhDs in the field!), the developments in the space have been evolving so quickly that it’s very hard to keep up with the pace of recent advancements and new techniques.
Moreover, many implementations of object detection techniques come from the academic world, where they were conceived as research prototypes. and not really intended to be used in production applications. This means that configuring them and modifying them can be challenging.
While there are some high level abstractions such as Keras, it is not always straightforward to apply them directly to the latest research findings. At the cost of simplicity, abstractions end up limiting what you can customize.
SaSS solutions are not always applicable
Many large companies have developed their own SaaS API solutions, like Google’s Vision API, Microsoft’s Computer Vision API, Amazon’s Rekognition, among others. Those are super simple to integrate into most applications and sometimes offer great solutions. However, these APIs usually only work for a predefined number of tasks/classes/object types like detecting faces, cars, dogs, cats, etc. While these are good for a broad range of applications, it is very common that companies have their own needs specific to the type of problem they are trying to solve. What if you are manufacturing company and want to identity defective pieces in your production chain? Or if you work in health care, and want to detect certain patterns in medical imagery? In these cases, you need something that can be trained with any dataset.
There are services that even let you upload samples and do the training in the cloud. However, sometimes you want to be the owner of your data model and deploy in your own servers, and not have an important part of your core business tech hosted outside your platform, using a Cloud API provider to whose model (or implementation) you have no real access to.
Besides, depending on the volume of usage you expect, SaaS might just become too expensive or the latency in transmitting images (which require high bandwidth) will be prohibitive for your application.
Creating a functional implementation from a research paper is damn hard
Even if you are an expert in any of the Deep Learning frameworks, coming up with something that actually works is pretty damn hard. Small implementation details have no room in academic papers, and sometimes they can and do make a huge difference in the results. Moreover, many things can be implemented in several ways, and these might in turn also affect your results.

Bridging the gap: birth of Luminoth

After identifying these issues and realizing that ourselves at Tryolabs ended up rewriting many of the common TensorFlow boilerplate code and models over and over for our clients, we started to think about the need for factoring out some code, and how an ideal toolkit would look like.
As a result, we built Luminoth, an open source toolkit for computer vision. Currently, we support object detection and image classification, but we are aiming for much more. It is built in Python, using TensorFlow, and Sonnet, a very useful library built by DeepMind for building complex neural networks with reusable components.

Luminoth’s main strengths

– State of the art algorithms: currently, we only support image classification and Faster R-CNN model for object detection, but we are actively working on providing more models and keeping up to date with the latest research in the field.
– Open source: Luminoth is free and open source. You can download it, customize it for your needs and integrate it into your product or cloud. You can also collaborate with it!
– Developer friendly: we’ve poured our experience of working on bridging the gap between academic Machine Learning findings and production ready software to make Luminoth accessible. We strived for an easy to use interface, beautiful code with comments, and unit tests. Of course, there is still a lot of room for improvement, and things will get better as the toolkit gets more mature.
– Made with TensorFlow and Sonnet: production ready, reliable, robust and maintained frameworks.
– Customizable: you can train Computer Vision models with your own data. You are not limited to existing datasets such as COCO or ImageNet.
– Cloud integration: we strived for a super simple Google Cloud integration, specifically ML Engine. This means training distributedly is very straightforward: there is no need to buy those GPUs.

Final thoughts

We expect Luminoth to be helpful both to developers wanting to integrate Deep Learning based computer vision algorithms into their products, as well as researchers who want to experiment with new models and techniques while skipping tedious boilerplate code.
Feel free to explore our GitHub repository. Feedback and contributions are more than welcome!

Alan is the CTO of Tryolabs. He holds a Computer Engineering degree and +7 yrs of experience developing robust backends, infrastructures and Machine Learning based algorithms. Currently, he’s the main consultant for every project developed and is an active member of the R&D team. A Python expert with deep understanding of Machine Learning related technologies.

Guest Post by Kanika Vatsyayant: Diving Into the Future of Deep Learning: Exploring the Novel Challenges

Deep Learning Infrastructure at Scale: An Overview

WHAT ARE THE CHALLENGES WITH SCALING UP DEEP LEARNING?

SOME OPEN SOURCE ML INFRASTRUCTURE SOLUTIONS