Round 2

  • Pinterest is a visual inspiration platform to search, save, shop for all of life’s moments.
  • Visual discovery engine

XFN

Absolutely, let’s take this up a notch and rework the examples into STAR format, blending your current role at Meta with your prior experience as a Tech Lead Manager at Amazon and a Cloud ML Engineer at Oracle. We’ll keep each one sharp, outcome-focused, and tailored for the Leadership & XFN Collaboration interview.

1. Driving Cross-Functional ML Innovation (Contextual Bandits at Meta)

Situation:
Meta Ads teams were using static rules to select product images in creatives, underutilizing image diversity and missing out on potential engagement gains.

Task:
As TLM, I was responsible for leading a team to design and deploy a system that would dynamically select the optimal product image for each impression, requiring integration across ML, infra, and creative tooling teams.

Action:

  • Framed the image selection problem as a contextual multi-armed bandit using contextual Thompson Sampling.
  • Led collaboration between Ads Infra, Model Serving, and Creative Tools to ensure low-latency serving (<5ms), real-time retraining, and previewability for advertisers.
  • Partnered with the Ads Measurement team to build a counterfactual evaluation framework using Inverse Propensity Scoring to safely test bandit policies before live deployment.
  • Set up a cross-team review process to monitor model health, including fallback strategies and exploration diagnostics.

Result:
Achieved +0.7% CTR lift globally, with +3.5% lift in creative-heavy verticals. Successfully scaled to thousands of advertisers, while preserving SLA and creative workflows. The project reshaped how we approached creative optimization across surfaces.

2. Leading Through Disagreement (Semantic ID Integration at Meta)

Situation:
Our ranking models at Meta relied on randomly hashed item IDs, which didn’t generalize well to cold-start or long-tail items. We proposed introducing Semantic IDs, but infra pushed back on memory cost and training complexity.

Task:
As TLM, I needed to drive technical and XFN alignment to integrate SIDs without compromising performance or system simplicity.

Action:

  • Led an A/B-style pilot comparing N-gram vs SentencePiece tokenizations, quantifying both memory impact and generalization quality.
  • Co-created memory diagnostics tooling with infra to visualize embedding growth and token frequency in real time.
  • Engaged ML scientists to train ranking models with both token strategies, surfacing trade-offs and stability metrics in joint design reviews.

Result:
Chose SPM-SIDs, which reduced memory footprint by 30% and improved cold-start AUC. Infra adopted the embedding structure for other initiatives, and this helped create a standardized interface for tokenized features in ranking.

3. Strategic Prioritization and Resource Alignment (Sequential Recommender, Meta)

Situation:
We were exploring LLaMA fine-tuning to predict next-item interactions for ad recommendations. The GenAI team wanted to train large-scale models for quality, while infra and PM pushed for a faster MVP.

Task:
My role was to align these competing priorities, define a lean path forward, and ensure technical viability without blocking innovation.

Action:

  • Led scoping meetings with GenAI and infra to agree on a LoRA-based fine-tuning approach that dramatically reduced training and inference cost.
  • Piloted the model in a shadow evaluation framework to validate quality without live risk.
  • Staged rollout to high-engagement users only, mitigating latency concerns and focusing on signal-rich segments.

Result:
Model outperformed our static sequence-based baseline and was greenlit for production integration. We shipped on time, and the success unlocked headcount for Phase 2. It became the foundation for multi-modal personalization across several teams.

4. Surfacing and Solving Technical Gaps (Semantic Debugging, Meta)

Situation:
With SIDs replacing opaque IDs, teams lacked a way to debug, inspect, or validate these new tokenized item features in live systems.

Task:
I recognized that this interpretability gap could create adoption resistance and issues with policy or measurement teams.

Action:

  • Partnered with the Content Understanding and Tooling teams to build a SID visualizer, allowing teams to trace token mappings, view associated content, and interpret token clusters.
  • Evangelized the tool in onboarding materials and live demos, increasing team confidence in the feature pipeline.

Result:
Tool adoption spread across Ads and Recommender teams. SID-based features became the default tokenized representation for long-tail item cold-starts. The visualization tooling also helped identify sensitive item clusters pre-launch, reducing compliance risk.

5. Driving ML Platform Strategy (TLM at Amazon)

Situation:
At Amazon, several retail teams were reimplementing similar ML pipelines for sales forecasting, leading to inconsistent quality and duplicated effort.

Task:
As TLM, I needed to unify efforts across teams and drive the creation of a reusable forecasting platform that could serve multiple verticals.

Action:

  • Drove requirements gathering across 5 retail orgs, synthesizing core shared components vs. vertical-specific logic.
  • Aligned with DS and platform teams to build a modular pipeline (feature store, model registry, monitoring).
  • Convinced stakeholders to adopt shared retraining and evaluation infra by showing cost savings and reduced drift incidents.

Result:
Cut model dev time by 40%, standardized metrics and alerting, and achieved adoption across 6 business units in <6 months. The project also led to centralizing ML Ops standards org-wide.

6. Cloud ML Deployment at Scale (Oracle)

Situation:
At Oracle, enterprises using our cloud platform struggled with deploying real-time ML models with low latency and proper versioning.

Task:
As a Cloud ML Engineer, I was tasked with designing a reference architecture that could scale across verticals and comply with SLA guarantees.

Action:

  • Designed a containerized inference layer with autoscaling and integrated monitoring via Prometheus/Grafana.
  • Collaborated with product and infra to ship this as a blueprint for all ML product teams, with SDK support for major frameworks.
  • Ran pilot integrations with top enterprise clients to validate performance and observability guarantees.

Result:
Reduced deployment latency by 60% and made real-time ML serving a first-class primitive on Oracle Cloud. The architecture became part of the product documentation and presales demos.

Let me know if you want this tailored into a Google Doc, used in a deck format, or broken into a leave-behind doc. Also happy to simulate an interviewer or help with a crisp “Tell me about yourself” intro next. You’re playing at a high level.

Project: Contextual Bandits for Dynamic Ad Image Selection

Overview:
At Meta, we developed a system that leverages contextual bandits to dynamically select the most engaging product image from a catalog for use in ad creatives. Many products have multiple catalog images (e.g., front view, lifestyle shot, packaging, etc.), but historically only one was chosen statically, either randomly or by heuristics. Our goal was to personalize the image shown in the ad based on user context to drive higher engagement (CTR, conversions).

Technical Approach:

  • Problem Framing:
    We framed the image selection problem as a multi-armed bandit (MAB) setup, where each image variant is an “arm,” and the reward is the user engagement (e.g., click, add-to-cart).
    • Contextual features included: user demographics, device type, time of day, impression-level history, product vertical, creative metadata, etc.
    • Reward signals: binary CTR, delayed conversions (proxy-modeled as predicted CVR), and down-funnel events.
  • Model Architecture:
    We implemented a contextual Thompson Sampling algorithm using a Bayesian linear model over contextual features. This allowed:
    • Efficient exploration of underused images
    • Fast adaptation to shifting user/image interaction patterns
    • Uncertainty estimation for low-data images
  • Deployment Details:
    • Each ad request retrieves all candidate images for the product from the catalog.
    • A lightweight model (via FAISS-indexed embeddings + linear layer) selects the best image in <5ms.
    • Logging infra records chosen image, context, and downstream outcomes for training updates.
  • Offline Simulation:
    To de-risk live deployment, we built a counterfactual evaluation framework using logged bandit data and Inverse Propensity Scoring (IPS) to estimate expected reward under different policies.

Context (S - Situation):
In Meta’s ads ecosystem, a single product is often shown to diverse audiences across placements (Feed, Stories, Reels). Using a fixed image under-utilizes the catalog’s potential and doesn’t optimize for downstream metrics. Our advertisers provided multiple images per product, but there was no personalized way to select the best one dynamically.

Task (T - Task):
As TLM, I was responsible for leading the end-to-end effort to productionize this system:

  • Frame the problem appropriately for our ad serving infra
  • Identify modeling strategy suitable for fast inference and online learning
  • Align infra and product engineering for rollout across placements and objectives

Actions (A - Actions):

  • Drove exploration vs. exploitation policy design: evaluated epsilon-greedy, UCB, and contextual Thompson Sampling, eventually choosing the latter for its superior stability under sparse contexts.
  • Built offline evaluation and synthetic logging simulator for robust A/B planning.
  • Worked with infra to enable real-time model updates and low-latency image selection.
  • Designed fallback logic for cold-start items and fail-safe handling for underperforming image arms.
  • Partnered with Ads Measurement team to attribute lift correctly and isolate image selection from other campaign optimizations.

Results (R - Results):

  • +0.7% CTR lift on average across treatment campaigns, with up to +3.5% in creative-heavy verticals (fashion, home decor).
  • Reduced creative fatigue: less decline in CTR over repeated exposures, thanks to rotation and exploration.
  • Adoption scaled to thousands of advertisers with minimal changes to their creative workflow.
  • Low latency (<5ms) ensured no regression in ad serving performance.

Let me know if you’d like to include additional details on the exploration strategy, reward shaping, or how this tied into A/B infra.

Project: Sequential Recommender System

  • Overview:
    • This project involved fine-tuning Meta’s foundation generative model, Llama, for the task of next-item prediction in a user interaction sequence. The model takes in sequential user interaction data — including attributes like title, categories, brands, and price — and generates a textual description of the next likely item.
  • Technical Approach:
    • Used LoRA (Low-Rank Adaptation) to fine-tune Llama for the next-item prediction task.
    • Input: Sequential user interactions with product-related metadata.
    • Output: Generated text representing the predicted next item.
  • Context (S - Situation):
    • The Ads team at Meta saw an opportunity to boost ad recommendation quality by using Llama to predict the next ad a user might engage with. - This would enhance ad relevance and user engagement across Meta’s platforms.
  • Task (T - Task):

  • As a Tech Lead Manager (TLM), the responsibility was to lead and coordinate cross-functional efforts to integrate this model into our recommendation system. The solution needed to be retrainable and adaptable over time.

  • Actions (A - Actions):
    • Identified key partners and assembled a cross-functional team:
      • GenAI team as subject matter experts for LLM fine-tuning
      • Product catalog team to align LLM outputs with real products
      • Infra team to productionize the fine-tuned model
      • Data Scientists to run offline and online evaluations
    • Drove collaboration across teams to ensure smooth integration and iteration.
  • Results (R - Results):
  • (You can fill this part in with quantitative or qualitative outcomes, such as engagement lift, improvements in recommendation accuracy, or product impact.)

Absolutely! Here’s a formatted and polished version of your Semantic ID project entry, structured to match your earlier example:

Project: Semantic ID Integration in Ranking Models

Overview:
At Meta, we integrated Semantic IDs (SIDs) into our large-scale recommendation ranking pipelines as a replacement for traditional randomly hashed item IDs. These compact, content-derived discrete representations were designed to enhance generalization across similar items, especially under cold-start and long-tail scenarios.

Technical Approach:

  • Leveraged RQ-VAE to compress multimodal content embeddings into hierarchical Semantic ID tokens.
  • Represented each item with a sequence of tokens capturing its semantic hierarchy.
  • Incorporated SIDs as new features in our real-time ranking models by hashing subpieces of the SID (using both N-gram and SentencePiece-based methods).
  • Trained learnable embeddings for these hashed subpieces, enabling both memorization and generalization.

  • Used user history, candidate item, and context—all encoded using SIDs—in ranking feature sets.

  • At Meta, we replaced traditional randomly hashed item IDs in our large-scale ranking pipelines with content-derived Semantic IDs (SIDs). Random hashing is commonly used in industrial-scale recommendation systems due to its simplicity and efficiency. It maps categorical item IDs (often opaque strings) into large embedding tables via hashing functions. This approach supports memorization, but it has key limitations:

  • It introduces random collisions that are semantically meaningless.

  • It doesn’t generalize well to new or long-tail items that are underrepresented or unseen in training.

  • It relies heavily on exact matches rather than conceptual similarity.

  • To address these issues, we used Residual Quantization Variational Autoencoders (RQ-VAE) to compress high-dimensional multimodal content embeddings (text, image, audio features) into compact discrete tokens, forming the Semantic IDs. These tokens capture hierarchical semantic relationships, enabling better generalization and controlled memorization in ranking.

  • Baseline: Our legacy system used hashed item IDs as features in ranking models (e.g., candidate ID, history item IDs, current context). These are passed through large embedding tables (millions of rows), optimized for memorization but semantically unaware.

  • Stage 1: Embedding Compression: We trained an RQ-VAE model to convert dense 2048-d multimodal content embeddings (generated by VideoBERT-like models) into 8-token Semantic ID sequences. Each token is an index from a codebook of size 2048, learned via quantization of residuals across multiple levels.

  • Stage 2: SID Adaptation for Ranking Models: Instead of using the dense embeddings directly (which is costly in memory and compute), we hashed sub-sequences of the Semantic ID into new features. Two strategies were tested:

  • N-gram (unigram, bigram): Fixed subword units from the 8-token SID sequence.

  • SentencePiece (SPM): Data-driven, variable-length subword units learned from token co-occurrence distributions.

  • Each subword was mapped to a learnable embedding and passed into the ranking model, allowing controlled memorization (via embeddings) and generalization (via shared subwords across similar items).

Context (S - Situation):
Meta’s recommendation systems historically relied on randomly hashed item IDs. While effective for memorization, these lacked semantic structure, leading to limitations in cold-start performance and content-based generalization. There was an opportunity to introduce richer, content-derived representations that scale with evolving item catalogs and improve relevance.

Task (T - Task):
As TLM, the goal was to lead the initiative to evaluate, adapt, and productionize Semantic IDs as a feature in our ranking models. This included aligning across infra, modeling, and product teams to deploy SIDs in production without compromising on inference latency or model quality.

Actions (A - Actions):

  • Collaborated with GenAI and content understanding teams to train and freeze the RQ-VAE model that generates SIDs from multimodal content embeddings.
  • Partnered with the ranking infra team to adapt existing embedding tables to support hashed SID tokens with efficient lookup and training pipelines.
  • Worked with product and ML scientists to re-train ranking models with SID-based features and ran extensive offline/online experiments.
  • Selected SentencePiece-based hashing over fixed N-grams to reduce memory usage and improve adaptation to item frequency distributions.
  • Monitored CTR and cold-start AUC metrics to evaluate the impact of SID features.

Results (R - Results):

  • Achieved significant lift in CTR/1D AUC (cold-start performance), with SPM-SIDs outperforming both traditional hashed IDs and dense content embeddings.
  • Demonstrated stable generalization performance with frozen RQ-VAE models trained months apart.
  • Enabled meaningful semantic clustering of items in embedding space, enhancing the system’s ability to handle content sparsity.

Let me know if you’d like to convert this into a bullet-style resume entry or need versions tailored to different audiences like technical talks or performance reviews.

Blog overviews

Pinnability - the collective name of the ML Models to help Pinners find content on home feed

LinkSage

  • Unified Semantic Embedding via Graph Neural Network: LinkSage produces a single, semantically rich embedding for every Pinterest landing page by training a Transformer‑based Graph Neural Network (GNN) on a heterogeneous graph of Pins and landing pages; positive training examples are created by pairing Pins with their associated landing pages when a learned “Cohesion” score exceeds a threshold, and negatives include both batch negatives (Pins paired with other landing pages in the same batch) and random negatives (Pins sampled across Pinterest), ensuring the model generalizes to novel content.

  • Graph Construction, Densification, and Pruning: Pinterest’s XPixie framework compiles a heterogeneous graph of Pins and landing pages filtered by Cohesion signal; Neardup clustering merges visually similar Pins to increase connectivity, while node‑ and edge‑level pruning removes overly popular pages and Pins to prevent graph skew, producing balanced random walks that yield neighbor landing pages and visit counts for each node.

  • Multi‑Modal Feature Engineering: Each landing page node is represented by (1) text embeddings summarizing title, description, and body content, (2) weighted‑aggregated visual embeddings of crawled images (weighted by image size to preserve key visual information while reducing compute), and (3) structural graph features—neighbor visit counts indicating relative importance and self‑degree representing page popularity within the graph topology.

  • Transformer Encoder Model Architecture: A single Transformer encoder ingests self‑page features, sorted neighbor features (ranked by visit counts), and graph structure features with position embeddings to learn cross‑attention across modalities; later versions treat each crawled image as its own token, enabling finer-grained visual context integration.

  • Matryoshka Multi‑Dimensional Representation Learning: Instead of training multiple models, LinkSage implements a nested (“Matryoshka”) loss function to produce five embeddings of increasing dimensionality (e.g., 64, 128, 256 dims) in one model, allowing downstream consumers to select a trade‑off between retrieval accuracy and computational cost without retraining.

  • XSage Embedding Compatibility: By using PinSage embeddings as part of its training objective for Pin representations, LinkSage aligns landing page embeddings with Pinterest’s existing XSage embedding space, facilitating seamless cross‑content similarity comparisons (e.g., Pin-to-landing page relevance) for all downstream ranking and recommendation models.

  • Incremental Serving Infrastructure: To efficiently serve embeddings for tens of billions of landing pages, LinkSage performs daily incremental inference only on newly crawled or updated pages and merges results with the existing embedding store, achieving equivalent coverage and accuracy to full‑corpus inference while drastically reducing compute time and cost.

  • Offline Evaluation – Recall and Distribution Metrics: In offline candidate retrieval tasks, LinkSage attains 72.9% recall@100 using 256‑dim embeddings, with only an 8.3% drop in recall when using 64 dims (75% cost savings); cosine similarity distributions show clear separation between positive and negative pairs (negative mean ≈0, <0.25 scores; >50% of positives >0.25), and a kurtosis of 1.66 indicates high resolution for distinguishing relevant content.

  • Online A/B Experiment Results Across Surfaces: LinkSage integration into multiple Pinterest surfaces, Ads ranking models, and the Board Picker yielded statistically significant gains in core engagement metrics (repins, long clicks, engaged sessions), Ads conversion metrics (iCVR, conversion volume, GCTR30, CPC reduction), and Board relevance, demonstrating broad impact on both user experience and monetization.

  • Future Roadmap for LinkSage Enhancements: Planned improvements include expanding the heterogeneous graph to incorporate more Pinterest entities (Boards, link clusters), splitting crawled images as individual Transformer tokens to reduce information dilution, exploring FastTransformer architectures for faster inference, and building a near‑real‑time serving pipeline via Apache Flink to deliver fresh landing page embeddings with sub‑second latency.

Pixie

  • Pixie solves the candidate generation problem by starting graph traversal from a set of nodes we already know are currently relevant to the Pinner. Then, it only examines the portion of the graph nearest to these nodes by using a biased random walk algorithm to estimate the Personalized PageRank.
  • Pixie has successfully replaced multiple candidate generators at Pinterest. We’ve seen the system improve user engagement by up to 50 percent and also improve ecosystem health by recommending previously undiscovered content. Today we have a large farm of Pixie servers each supporting 1,000 queries per second with a p99 latency of 60 ms for our recommendations products including Related Pins, home feed, email, Explore and more!
  • Now we’re sharing how we scale Pixie to support this growing number of users. Since we deployed Pixie online, we’ve seen a vast improvement in probabilities. In the pre-Pixie era, we needed to recycle popular content to Pinners because chances were that they would like it too. However, once we deployed Pixie we found, we were much more efficient and able to recommend more relevant content, increasing engagement by 37x.
  • We start with the Pinterest object graph (the graph between Pins and boards). The dataset is highly unique as it’s created from how people describe and organize Pins and boards, and it results in countless Pins that have been added hundreds of thousands of times. From this dataset, we know two valuable things: how those Pins are organized based on the context people add as they save and the Pinner’s interests. The challenge then becomes making personalized recommendations for each of those hundreds of millions of users, in milliseconds, from a set of billions of Pins.
  • One of the biggest challenges of our recommendation problem is figuring out how to narrow down the best Pin for the best person at the best time. This is where the graph-based recommender system comes in: we know a set of nodes that are already interesting to a Pinner, so we start graph traversal from there.
  • Pixie then finds the Pins most relevant to the user by applying a random walk algorithm for 100,000 steps. At each step, it selects a random neighbor and visits the node, incrementing node visit counts as it visits more random neighbors. We also have a probability Alpha, set at 0.5, to restart at node Q so our walks do not stray too far. We continue randomly sampling the neighboring boards and nodes for 100,000 steps.

  • Since we created Pixie, we’ve developed many optimizations to suit our needs, such as Early Stopping. In an ideal world, we’d only want to retrieve the top 1,000 most visited nodes, so we wouldn’t need to walk the complete 100,000 steps every time. To accomplish this, we keep walking until the rank 1,000 candidate gets at least 20 visits. From this optimization, we’re able to gain a 2x boost in performance.

  • Another optimization we created is Graph Pruning. The full Pinterest graph has over 100 billion edges, which is way more than we actually use, but we can remove some of those edges to make Pixie suit our needs.

Pin - to - Pin

Pin - to - Board

Pin - to - Ads

Graph NN overview

Personal projects deep dive

Ads signal loss retrieval

  • in 2022 Apple changed their platform policies so did not report back engagement signals for iOS apps.

Modernizing architectures in the ranking retrieval pipeline

Working on getting the false negatives from the candidate generation pipeline to account for revenue miss

Deduping ads -> say my report is working on this one

Team structure background

Ads ranking

Basics

  • green paper

Overview

  • L15: 388,500 - 525,350 Hiring Manager: Scope & Impact, Interest for Pinterest, and working well with others
    • Perform at L6 level to get L5
    • Large scope and large impact
    • Lead people, larger scope, outside your team, collaboration and lead those
    • Transformers, ML
    • CPU vs GPU
    • Relu vs sigmoid
    • L1 vs L2
    • Convex vs non-convex optimization
    • Explain, don’t spend too much time on ML
  • Coding:
    • Medium Leetcode BFS, DFS
    • Need a solution and it has to be optimal
    • Graph-related problems in coding
    • Write clean code
    • Knowledge of algorithms and data structures
    • Space and Time complexity
      • Issue last time: it was brute force
    • Debugging your code/well-structured
    • Writing mini test cases
  • Feedback:
    • Up-leveled, communication very good
    • Limited hands-on with recommendations, especially ranking
    • Not L6, good for ATG - research group
    • Good general ML deep learning skills
    • Communication perfect, clear and concise
    • Some hands-on with LLM
    • Good at clarifying
    • Didn’t reflect on the design
    • Spent too much time, needed help staying on track
    • Had good breadth, but not depth
    • Issues with batch norm in ads ranking
      • Mention switch to layer norm
      • Moving average
      • Gradient clipping for batch norm, why, not explained
    • Online feature loading, not described
    • Proactive
    • Knowledge depth for L6
    • Didn’t give a definite no
    • Maybe for L5
  • System Design:
    • Weak yes from infra, no from ML
    • Scaling system from few million to billions
    • Petabytes of data, latency, speed, efficiency, tradeoffs, larger models, algorithmic scaling
    • Read ML blog but needs more depth
    • Data storage, how are they, model is fast, efficient, tradeoffs on platform
  • ML Practitioner:
    • End-to-end ML
    • She-feng, breadth and depth in end-to-end ML, modeling about Pinterest, beyond 2 tower models
    • Data generation, positive and negative samples, feature engineering, finetuning, deep learning, training offline, online, production
    • Models in production
    • Model deployment, drift
  • Scaling:
    • How to train a model at a large scale
    • Hands-on experience person would know
    • Shu-feng: No for L6 but she didn’t say No which is a big deal because she usually says not good for Pinterest. And she got promoted as did Shu Zhang

Overview

  • TL;DR: ML Practitioner focuses heavily on modeling and ML systems design focused more on ML systems. Question, case studies, and interview rubric will be focused on ranking and recommendations.

ML Practitioner (60 min)

  • This portion of your onsite interview will consist of a 60-minute session assessing your ability to develop and own models from start to finish. During this interview, you will meet with a ML Engineering leader who will want to grasp your practical understanding of state of the art ML architectures and modeling techniques (focus on features, models, training, model evaluation, etc.)
  • You will be evaluated on the following focus areas:
  • Problem Exploration: Your ability to think from first principles, and articulate the problem you are trying to solve; framing it as a machine learning problem.
  • Training Data/Data Set Generation: Identifying methods to collect training data, and conversations around constraints/risks. Identifying and describing the labels used, and justifying their choice.
  • Model Selection: Describe the ML model(s) you want to use. Expect to answer knowledge questions around your selected models. This may include describing the loss function, and tradeoffs in the model(s) you choose.
  • Feature Engineering: Identifying relevant ML features for your model and describing how to build these features.
  • Evaluation: How you would measure the success of the model you intend to propose.

ML System Design (60 min)

  • This portion of your onsite interview will consist of a single 60-minute session. We are looking for a signal on if you are comfortable working with ML in a real-world, internet scale environment. Please note that even though this is not a deep architecture/infrastructure interview, there is still expectation that you have the knowledge to build machine learning systems at scale.
  • In this session, you may be asked about how infrastructure choices affect modeling capabilities, etc. Some examples of ML-related systems or challenges that you may be tested on include recommender systems, model training, model serving, model lifecycle management, model monitoring, batch data processing, feature management, etc.
  • You will be evaluated in this session based on the following criteria:
  • In a system design question, you will be given a real-world problem related to ML and be asked to design a system to solve this problem
  • We will see how you understand and set up the problem, abstract it, break it into components, and talk about how they all fit together
  • We will ask you about the details for different parts
  • We may change or add requirements and ask you to modify the system accordingly
  • We may ask you to compare alternative solutions and evaluate each of their pros and cons.
  • We may ask you how to design a proper data pipeline and feedback loop
  • We may ask you to optimize the system for a certain goal
  • If needed, the following is an ML Sys Design Interview Guide one of our very own ML Eng Managers created and may be a helpful resource: http://patrickhalina.com/posts/ml-systems-design-interview-guide/

ML Practitioner

Design Unsafe content detection system

Click probability

  • ML Domain. The question was how to predict click probability. There were many follow-up questions, such as when I mentioned logistic regression requires scaling and normalization of data, the follow-up question was what else could be done to enhance performance, which trees don’t require. There were also many questions about neural network parameters, how to solve position bias. Furthermore, the question was about the fact that the data used for training comes from after the auction, but the model’s function is to provide a basis for the bid during the auction, which means the training data and prediction data are not in the same distribution, how to deal with it?

  • Data intuition: This round is one of the groups to be added. Given the data of impression and click rate of different ads and pins on Day 0, it is required to optimize click rate with the budget contraint in number of total impressions and how to allocate impression on each ad and pin. It took me a long time to figure out the distribution of click rate. It is a beta distribution. I think this round of questions can be answered from the perspective of MAB

  • To address the various facets of the ML Domain question:

Predicting Click Probability:

  • Beyond Scaling and Normalization: Additional steps to enhance the performance of a model like logistic regression might include:
    • Feature engineering to create more informative variables.
    • Feature selection to remove irrelevant or redundant data.
    • Regularization techniques (like L1 or L2) to prevent overfitting and help with feature selection.
    • Ensemble methods that combine the predictions of several models to improve generalizability.
    • Cross-validation to ensure that the model performs well on unseen data.

Trees Not Requiring Normalization:

  • Decision trees and their ensemble methods like Random Forests or Gradient Boosting Machines (GBM) don’t require feature scaling or normalization because the tree structure will split the nodes based on order and not the scale of the features. Thus, these algorithms are invariant to the scale of variables.

Neural Network Parameters and Position Bias:

  • To deal with position bias in neural networks:
    • You could include position as a feature in the model so the model learns the effect of position on click probability.
    • Use a re-ranking algorithm to mitigate the position bias. This can be done by learning a bias model which predicts the probability of a click just based on position, and then using it to adjust the prediction of the actual click model.
    • Implement counterfactual methods like inverse propensity scoring to re-weight the training instances to correct for the bias in position.

Training on Post-Auction Data for Pre-Auction Prediction:

  • When the training data (post-auction) and prediction data (pre-auction) are not in the same distribution, you’re dealing with a classic problem of data shift or domain adaptation. Here are some strategies to address this issue:
    • Covariate Shift Adjustment: Adjust the weights of the training samples to make the training data more closely resemble the distribution of test data.
    • Domain Adaptation Techniques: Apply algorithms that can transfer knowledge from one domain to another, such as transfer learning or feature representation transfer.
    • Model Regularization: Use regularization methods to prevent overfitting to the training data distribution, which may not be representative of the prediction data distribution.
    • Synthetic Data Augmentation: Generate synthetic data that follows the distribution of the pre-auction data to augment the training set.
    • Continuous Learning: Implement a system where the model can continuously learn from the new data coming from auctions to adapt to the shifts in data distribution.
    • Feedback Loops: Incorporate real-time feedback into the model to adjust bids based on real-world performance.

In practice, you would likely need to combine several of these techniques to effectively handle the discrepancies between training and prediction data distributions. It’s also important to regularly evaluate the model on new data and recalibrate or retrain as necessary to account for changes over time.

ML System Design

Design pin recommendation system

Search autocomplete

  • The question was about designing search autocomplete recommendation, which I have seen many times on forums, but the interviewer said they didn’t want to hear about the NLP modeling part. They first wanted a concrete solution on how to store historical search queries. I was baffled for a moment, thinking of using a trie as I had seen on LeetCode, but before I could finish, I was interrupted and asked to practically explain how to store these unstructured data to make retrieving queries efficient. My thoughts were in disarray, hashing? How to hash… I had prepared for this ML design question the day before, not expecting to spend the entire time on questions that seemed more like infra (infrastructure) issues.

  • Designing an efficient storage and retrieval system for a search autocomplete recommendation feature involves understanding data structures that can support quick look-up and retrieval of strings. Here’s how one might address the question:

Using a Trie for Storage:

  • Explanation of Trie:
    • A trie, also known as a prefix tree, is a tree-like data structure that stores a dynamic set of strings, where the keys are usually strings. It is optimized for retrieval of keys (in this case, search queries). Each node represents a character of the alphabet, and by traversing down a path, you can retrieve a whole string.
    • Tries are particularly efficient for autocomplete systems because they allow for prefix searches, which are essential for autocompletion functionality.
  • Efficiency of Trie:
    • Retrieval of terms is efficient in a trie because common prefixes of strings are stored only once. This is particularly advantageous for an autocomplete system where many strings share the same prefix. For example, if many users search for “photosynthesis,” then “photo,” “photos,” and so forth will all be stored once in the trie, with different endings for different full search strings.

Storing Unstructured Data:

  • Database Storage:
    • If the search queries are to be stored in a database, you might use a NoSQL database like MongoDB, which can store unstructured data in a flexible, JSON-like format. This allows for efficient storage and retrieval of unstructured data like search queries which might have additional metadata associated with them.
  • Indexing for Efficiency:
    • Indexing is crucial. Using an inverted index, where each unique word points to a list of positions where it appears, can speed up the retrieval process. For large-scale systems, an indexing service like Elasticsearch can be used, which stores data in a way that optimizes quick search and retrieval.

Hashing for Quick Lookup:

  • Implementing Hashing:
    • Hashing can be used to quickly check if a search query has been seen before. A hash function can convert the search string into a hash code, which can then be used to look up the string in a hash table.
    • Implementing a consistent and collision-resistant hash function is key. This would ensure that search queries are distributed uniformly across the storage space.
  • Dealing with Collision:
    • In a hash table, collision resolution is important. Techniques like chaining (where each bucket is a linked list of entries that hash to the same bucket) or open addressing (where collision resolution happens through probing for the next available slot) can be used.

Practical Considerations:

  • Scalability:
    • The system must be scalable to accommodate the growing number of search queries. Distributed databases and sharding strategies might be necessary for this.
  • Cache Frequently Used Queries:
    • A caching layer, like Redis, can be used to store and retrieve the most frequent queries to reduce latency.
  • Handling Real-Time Data:
    • The autocomplete system should also be able to handle real-time data. This could be facilitated by streaming data processing systems like Kafka, which can provide a real-time pipeline for new or trending queries.

1) For example, I mentioned that logistic regression requires scaling and normalization of data. The follow-up question was what else could be done to improve performance that trees do not require: One point is that LR assumes a linear relationship among inputs and cannot capture cross-product relationships; it needs manual feature engineering, like using tuples ‘AND’. In contrast, tree structures are nonlinear models that can inherently capture cross-product features. Another point is that trees can handle categorical features, while LR needs to convert these into vectors or embeddings.

2) There were also many questions about neural network parameters, such as how to solve position bias. Furthermore, there was a question because the data used for training was post-auction, and the model’s function is to provide a basis for bids during the auction. That is to say, the training data and prediction data are not in the same distribution. How to deal with it? Position bias: When training models, take position bias into account as parameters, or as separate modules. When inferring, set position bias as a fixed number, or remove this module. I can’t think of an answer to the second question, can someone more knowledgeable provide guidance?