One of many largest challenges I encountered in my profession as a knowledge scientist was migrating the core algorithms in a cellular AdTech platform from basic machine studying fashions to deep studying. I labored on a Demand Aspect Platform (DSP) for consumer acquisition, the place the function of the ML fashions is to foretell if exhibiting advert impressions to a tool will outcome within the consumer clicking on the advert and putting in a cellular app. For a fast hands-on overview of the clicking prediction drawback, please try my past post.
Whereas we have been in a position to shortly get to a state the place the offline metrics for the deep studying fashions have been aggressive with logistic regression fashions, it took awhile to get the deep studying fashions working easily in manufacturing, and we encountered many incidents alongside the way in which. We have been in a position to begin with small-scale checks utilizing Keras for mannequin coaching and Vertex AI for managed TensorFlow serving, and ran experiments to check iterations of our deep studying fashions with our champion logistic regression fashions. We have been finally in a position to get the deep studying fashions to outperform the basic ML fashions in manufacturing, and modernize our ML platform for consumer acquisition.
When working with machine studying fashions on the core of a posh system, there are going to be conditions the place issues are going to go off the rails and it’s necessary to have the ability to shortly get better and be taught from these incidents. Throughout my time at Twitch, we used the Five W’s approach to writing postmortems for incidents. The concept is to establish “what” went improper, “when” and “the place” it occurred, “who” was concerned, and “why” an issue resulted. The comply with up is to then set up learn how to keep away from the sort of incident sooner or later and to arrange guardrails to forestall related points. The objective is to construct a increasingly sturdy system over time.
In certainly one of my previous roles in AdTech, we bumped into a number of points when migrating from basic ML fashions to deep studying. We finally bought to a state the place we had a sturdy pipeline for coaching, validating, and deploying fashions that improved upon our basic fashions, however we bumped into incidents throughout this course of. On this put up we’ll cowl 8 of the incidents that occurred and describe the next steps we took for incident administration:
- What was the difficulty?
- How was it discovered?
- How was it fastened?
- What did we be taught?
We recognized a wide range of root causes, however typically aligned on related options when making our mannequin pipelines extra sturdy. I hope sharing particulars about these incidents supplies some steering on what can go improper when utilizing deep studying in manufacturing.
Incident 1: Untrained Embeddings
What was the difficulty?
We discovered that lots of the fashions that we deployed, comparable to predicting click on and set up conversion, have been poorly calibrated. This meant that the expected worth of conversion by the mannequin was a lot larger than the precise conversion that we noticed for impressions that we served. After drilling down additional, we discovered that the miscalibration was worse on categorical options the place we had sparse coaching knowledge. Ultimately we found that we had embedding layers in our set up mannequin the place we had no coaching knowledge obtainable for a number of the vocabulary entries. What this meant is that when becoming the mannequin, we weren’t making any updates to those entries, and the coefficients remained set to their randomized initialized weights. We known as this incident “Untrained Embeddings”, as a result of we had embedding layers the place a number of the layer weights by no means modified throughout mannequin coaching.
How was it discovered?
We largely found this subject via instinct after reviewing our fashions and knowledge units. We used the identical vocabularies for categorical options throughout two fashions, and the set up mannequin knowledge set was smaller than the clicking mannequin knowledge set. This meant that a number of the vocabulary entries that have been wonderful to make use of for the clicking mannequin have been problematic for the set up mannequin, as a result of a number of the vocab entries didn’t have coaching examples within the smaller knowledge set. We confirmed that this was the difficulty by evaluating the weights within the embedding layers earlier than and after coaching, and discovering {that a} subset of the weights have been unchanged after becoming the mannequin. As a result of we randomly initialized the weights in our Keras fashions, this led to points with the mannequin calibration on reside knowledge.
How was it fastened?
We first restricted the scale of our vocabularies used for categorical options to scale back the probability of this subject occurring. The second change we made was setting the weights to 0 for any embedding layers entries the place the weights have been unchanged throughout coaching. Long term, we moved away from reusing vocabularies throughout totally different prediction duties.
What did we be taught?
We found that this was one of many points that was resulting in mannequin instability, the place fashions with related efficiency on offline metrics would have noticeably totally different efficiency when deployed to manufacturing. We ended up constructing extra tooling to check mannequin weights throughout coaching runs as a part of our mannequin validation pipeline.
Incident 2: Padding Challenge with Batching for TensorFlow Serving
What was the difficulty?
We migrated from Vertex AI for mannequin serving to an in-house deployment of TensorFlow serving, to cope with a number of the tail-end latency points that we have been encountering with Vertex on the time. When making this variation, we bumped into a difficulty with learn how to cope with sparse tensors when enabling batching for TensorFlow serving. Our fashions contained sparse tensors for options, such because the listing of recognized apps put in on a tool, that may very well be empty. Once we enabled batching when serving on Vertex AI, we have been in a position to make use of empty arrays with out subject, however for our in-house mannequin serving we bought error responses when utilizing batching and passing empty arrays. We ended up passing “[0]” values as an alternative of “[ ]” tensor values to keep away from this subject, however this once more resulted in poorly calibrated fashions. The core subject is that “0” referred to a particular app fairly than getting used for out-of-vocab (OOV). We have been introducing a characteristic parity subject to our fashions, as a result of we solely made this variation for mannequin serving and never for mannequin coaching.
How was it discovered?
As soon as we recognized the change that had been made, it was easy to show that this padding strategy was problematic. We took data with an empty tensor and adjusted the worth from “[]” to “[0]” whereas preserving the entire different tensor values fixed, and confirmed that this variation resulted in numerous prediction values. This made sense, as a result of we have been altering the tensor knowledge to say that an app was put in on the machine the place that was not really the case.
How was it fastened?
Our preliminary repair was to alter the mannequin coaching pipeline to carry out the identical logic that we carried out for mannequin serving, the place we change empty arrays with “[0]”, however this didn’t fully tackle this subject. We later modified the vocab vary from [0, n-1] to [0, n], the place 0 had no which means and was added to each tensor. This meant that each sparse tensor had not less than 1 worth and we have been in a position to make use of batching with our sparse tensor setup.
What did we be taught?
This subject largely got here up resulting from totally different threads of labor on the mannequin coaching and mannequin serving pipelines, and lack of coordination. As soon as we recognized the variations between the coaching and serving pipelines, it was apparent that this discrepancy may trigger points. We labored to enhance on this incident by together with knowledge scientists as reviewers on pull requests on the manufacturing pipeline to assist establish a lot of these points.
Incident 3: Untrained Mannequin Deployment
What was the difficulty?
Early on in our migration to deep studying fashions, we didn’t have many guardrails in place for mannequin deployments. For every mannequin variant we have been testing we’d retrain and mechanically redeploy the mannequin every day, to make it possible for the fashions have been skilled on latest knowledge. Throughout one of many coaching runs, the mannequin coaching resulted in a mannequin that at all times predicted a 25% click on fee whatever the enter knowledge and the ROC AUC metric on the validation knowledge set was 0.5. We had primarily deployed a mannequin to manufacturing that at all times predicted a 25% click on fee no matter any of the characteristic inputs.
How was it discovered?
We first recognized the difficulty utilizing our system monitoring metrics in Datadog. We logged our click on predictions (p_ctr) as a histogram metric, and Datadog supplies p50 and p99 aggregations. When the mannequin was deployed, we noticed the p50 and p99 values for the mannequin converge to the identical worth of ~25%, indicating that one thing had gone improper with the clicking prediction mannequin. We additionally reviewed the mannequin coaching logs and noticed that the metrics from the validation knowledge set indicated a coaching error.
How was it fastened?
On this case, we have been in a position to rollback to the clicking mannequin from yesterday to resolve the difficulty, nevertheless it did take a while for the incident to be found and our rollback strategy on the time was considerably guide.
What did we be taught?
We discovered that this subject with unhealthy mannequin coaching occurred round 2% of the time and wanted to arrange guardrails towards deploying these fashions. We added a mannequin validation module to our coaching pipeline that checked for thresholds on the validation metrics, and in addition in contrast the outputs of the brand new and prior runs on the mannequin on the identical knowledge set. We additionally arrange alerts on Datadog to flag giant modifications within the p50 p_ctr metric and labored on automating our mannequin rollback course of.
Incident 4: Unhealthy Warmup Information for TensorFlow Serving
What was the difficulty?
We used warmup files for TensorFlow serving to enhance the rollout time of recent mannequin deployments and to assist with serving latency. We bumped into a difficulty the place the tensors outlined within the warmup file didn’t correspond to the tensors outlined within the TensorFlow mannequin, leading to failed mannequin deployments.
How was it discovered?
In an early model of our in-house serving, this mismatch between warmup information and mannequin tensor definitions would trigger all mannequin serving to return to a halt and require a mannequin rollback to stabilize the system. That is one other incident that was initially captured by our operational metrics on Datadog, since we noticed a big spike in mannequin serving error requests. We confirmed that there was a difficulty with the newly deployed mannequin by deploying it to Vertex AI and confirming that the warmup information have been the basis reason behind the difficulty.
How was it fastened?
We up to date our mannequin deployment module to verify that the mannequin tensors and warmup information have been suitable by launching an area occasion of TensorFlow serving within the mannequin coaching pipeline and sending pattern requests utilizing the warmup file knowledge. We additionally did extra guide testing with Vertex AI when launching new kinds of fashions with noticeably totally different tensor shapes.
What did we be taught?
We realized that we wanted to have totally different environments for testing TensorFlow mannequin deployments earlier than pushing them to manufacturing. We have been in a position to do some testing with Vertex AI, however finally arrange a staging setting for our in-house model of TensorFlow serving to supply a correct CI/CD setting for mannequin deployment.
Incident 5: Problematic Time-Based mostly Options
What was the difficulty?
We explored some time-based options in our fashions, comparable to weeks_ago, to seize modifications in habits over time. For the coaching pipeline, this characteristic was calculated as flooring(date_diff(at this time, day_of_impression)/7). It was a extremely ranked characteristic in a few of our fashions, nevertheless it additionally added unintended bias to our fashions. Throughout mannequin serving, this worth is at all times set to 0, since we’re making mannequin predictions in actual time, and at this time is similar as day_of_impression. The important thing subject is that the mannequin coaching pipeline was discovering patterns within the coaching knowledge which will create bias points when making use of the mannequin on reside knowledge.
How was it discovered?
This was one other incident that we discovered largely via instinct and later confirmed to be an issue by evaluating the implementation logic throughout the coaching and mannequin serving pipelines. We discovered that the mannequin serving pipeline at all times set the worth to 0 whereas the coaching pipeline used a variety of values provided that we frequently use months previous examples for coaching.
How was it fastened?
We created a variant of the mannequin with the entire relative time-based options eliminated and did an A/B take a look at to check the efficiency of the variants. The mannequin that included the time based mostly options carried out higher on the holdout metrics throughout offline testing, however the mannequin with the options eliminated labored higher within the A/B take a look at and we ended up eradicating the options from the entire fashions.
What did we be taught?
We discovered that we had launched bias into our fashions in an unintended approach. The options have been compelling to discover, as a result of consumer habits does change over time, and introducing these options did lead to higher offline metrics for our fashions. Ultimately we determined to categorize these as problematic below the characteristic parity class, the place we see variations in values between the mannequin coaching and serving pipelines.
Incident 6: Suggestions Options
What was the difficulty?
We had a characteristic known as clearing_price that logged how excessive we have been prepared to bid on an impression for a tool over the past time that we served an advert impression for the machine. This was a helpful characteristic, as a result of it helped us to bid on units with a excessive bid flooring, the place the mannequin wants excessive confidence {that a} conversion occasion will happen. This characteristic by itself typically wasn’t problematic, nevertheless it did change into an issue throughout an incident the place we launched unhealthy labels into our coaching knowledge set. We ran an experiment that resulted in false positives in our coaching knowledge set, and we began to see a suggestions subject the place the mannequin bias turned a difficulty.
How was it discovered?
This was a really difficult incident to establish the basis reason behind, as a result of the experiment that generated the false constructive labels was run on a small cohort of visitors, so we didn’t see a sudden change in operational metrics like we did with a number of the different incidents in Datadog. As soon as we recognized which units and impressions have been impacted by this take a look at, we regarded on the characteristic drift of our knowledge set and located that the typical worth of the clearning_price characteristic was rising steadily for the reason that rollout of the experiment. The false positives within the label knowledge have been the basis reason behind the incident, and the drift on this characteristic was a secondary subject that was inflicting the fashions to make unhealthy predictions.
How was it fastened?
Step one was to rollback to a best-known mannequin earlier than the problematic experiment was launched. We then cleaned up the information set and eliminated the false positives that we may establish from the coaching knowledge set. We continued to see points and in addition made the decision to take away the problematic characteristic from our fashions, much like the time-based options, to forestall this characteristic from creating future suggestions loops sooner or later.
What did we be taught?
We realized that some options are useful for making the mannequin extra assured in predicting consumer conversions, however will not be definitely worth the threat as a result of they will introduce a tail-spin impact the place the fashions shortly deteriorate in efficiency and create incidents. To interchange the clearing value characteristic, we launched new options utilizing the minimal bid to win values from auction callbacks.
Incident 7: Unhealthy Characteristic Encoding
What was the difficulty?
We explored a number of options that have been numeric and computed as ratios, comparable to the typical click on fee of a tool, computed because the variety of clicks over the variety of impressions served to the machine. We ran right into a characteristic parity subject the place we dealt with divide by zero in numerous methods between the coaching and serving mannequin pipelines.
How was it discovered?
We now have a characteristic parity examine the place we log the tensors created throughout mannequin inference for a subset of impressions and run the coaching pipeline on these impressions and evaluate the values generated within the coaching pipeline with the logged worth at serving time. We seen a big discrepancy for the ratio based mostly options and located that we encoded divide by zero as -1 within the coaching pipeline and 0 within the serving pipeline.
How was it fastened?
We up to date the serving pipeline to match the logic within the coaching pipeline, the place we set the worth to -1 when a divide by zero happens for the ratio based mostly options.
What did we be taught?
Our pipeline for detecting characteristic parity points allowed us to shortly establish the basis reason behind this subject as soon as the mannequin was deployed to manufacturing, nevertheless it’s additionally a state of affairs we need to keep away from earlier than a mannequin is deployed. We utilized the identical studying from incident 2, the place we included knowledge scientists on pull request opinions to assist establish potential points between our coaching and serving mannequin pipelines.
Incident 8: String Parsing
What was the difficulty?
We used a 1-hot encoding strategy the place we select the highest ok values, that are assigned indices from 1 to ok, and use 0 as an out-of-vocab (OOV) worth. We bumped into an issue with the encoding from strings to integers when coping with categorical options comparable to app bundle, which regularly has extra characters. For instance, the vocabulary might map the bundle com.dreamgames.royalmatch to index 3, however within the coaching pipeline the bundle is about to com.dreamgames.royalmatch$hl=en_US and the worth will get encoded to 0, as a result of it’s thought of OOV. The core subject we bumped into was totally different logic for sanitizing string values between the coaching and serving pipelines earlier than making use of vocabularies.
How was it discovered?
This was one other incident that we found with our characteristic parity checker. We discovered a number of examples the place one pipeline encoded the values as OOV whereas the opposite pipeline assigned non-zero values. We then in contrast the characteristic values previous to encoding and seen discrepancies between how we did string parsing within the coaching and serving pipelines.
How was it fastened?
Our brief time period repair was to replace the coaching pipeline to carry out the identical string parsing logic because the serving pipeline. Long term we centered on truncating the app bundle names on the knowledge ingestion step, to scale back the necessity for guide parsing steps within the totally different pipelines.
What did we be taught?
We realized that coping with problematic strings at knowledge ingestion supplied essentially the most constant outcomes when coping with string values. We additionally bumped into points with unicode characters exhibiting up in app bundle names and labored to appropriately parse these throughout ingestion. We additionally discovered it crucial to often examine the vocabulary entries which might be generated by the system to verify particular characters weren’t exhibiting up in entries.
Takeaways
Whereas it could be tempting to make use of deep studying in manufacturing for mannequin serving, there’s a variety of potential points which you can encounter with reside mannequin serving. It’s necessary to have sturdy plans in place for incident administration when working with machine studying fashions, in an effort to shortly get better when mannequin efficiency turns into problematic and be taught from these missteps. On this put up we coated 8 totally different incidents I encountered when utilizing deep studying to foretell click on and set up conversion in a cellular AdTech platform. Right here’s are the important thing takeaways I realized from these machine studying incidents:
- It’s necessary to log characteristic values, encoded values, tensor values, and mannequin predictions throughout mannequin serving, to make sure that you wouldn’t have characteristic parity or mannequin parity points in your mannequin pipelines.
- Mannequin validation is a crucial step in mannequin deployment and take a look at environments may help cut back incidents.
- Watch out for the options that you just embody in your mannequin, they could be introducing bias or inflicting unintended suggestions.
- You probably have totally different pipelines for mannequin coaching and mannequin serving, the staff members engaged on the pipelines must be reviewing one another’s pull requests for ML characteristic implementations.
Machine studying is a self-discipline that may be taught quite a bit from DevOps to scale back the incidence of incidents, and MLOps ought to embody processes for effectively responding to points with ML fashions in manufacturing.