Overview

Tech Breadth

  • General Machine Learning Knowledge: Questions will focus on ML basic concepts, ML metrics, data handling, deep learning, etc.
    • Note: We’re NOT looking for expertise in all areas; rather, a general understanding of practical ML concepts. Thorough understanding and knowledge depth will be a bonus.

  • Question: “How can we test the effectiveness of a rider incentive program when there is a shortage of riders? Is this a case for A/B testing?”

  • Answer: Yes, you can test the effectiveness of a rider incentive program using A/B testing, even if there is a shortage of riders. Here’s how you can set it up:

  1. Define the Objective: Clearly state what you aim to achieve with the incentive program, such as increasing the number of rides or improving rider retention.

  2. Segment Your Riders: Randomly divide your riders into two groups:
    • Group A (Control Group): This group will not receive any incentives.
    • Group B (Test Group): This group will receive the incentives.

  3. Run the Test: Implement the incentive program for Group B while keeping Group A unchanged.

  4. Measure the Results: Compare the performance metrics (e.g., number of rides, rider retention rates) between the two groups after a specified period.

  5. Analyze the Data: Determine if there is a statistically significant difference between the two groups. If Group B shows improvement over Group A, the incentive program can be considered effective.

Sys Des

  • DoorDash blog

  • “Design a review system where users can review a dish (not including reviews of restaurants), view all reviews of a dish, view all their own reviews, and give a review a thumbs up or thumbs down. Then, based on the user’s input.”

  • System design: Design restaurant ranking. Half of the discussion is about geohash/R-tree. The other half is about ranking end-to-end pipeline. I have done search ranking so the second part is faster.”

  • design a system similar to a Slack chat

  • “1bh: Asked how to resolve conflicts, diversity if there are non-Asian people, women, etc., in your team. How do you get along with them? Focused on asking if the restaurant makes mistakes such as missing menu, wrong number of items entered, etc. How to deal with these mistakes that are not caused by DD.

  • 2 system design: Instagram 2b user focused on uploading and downloading pictures and videos, storage and architecture design. The client needs to do compression, split by chunk, cache, etc.”

  • DoorDash’s question is to predict the final delivery time, which is a regression problem. It is completely different from other companies’ DS questions. In addition, the data volume is relatively large and the data is relatively messy. It is suitable for everyone to practice clean data.

  • Design a review system where users can review a dish (not including reviews of restaurants), view all reviews of a dish, view all their own reviews, and give a review.

  • Areviewsystemwhereuserscanwritereviewsreadreviewslikeanddislike

  • The domain knowledge check tested 1) the technical knowledge written on the resume, 2) asked how these technologies were used in the project. The review system is the design Amazon review system, which allows you to review the products you bought, and other customers can upvote/downvote you.

Here’s a more detailed outline for the system design of a recommender system specifically tailored for ranking restaurants, incorporating both a nuanced understanding of machine learning components and the specifics of implementation:

System Design Recommender System Outline

  1. Framing the Problem as ML: Design a Restaurant Ranking System
    • Objective: Develop a system that ranks restaurants based on individual user preferences and contextual data.
    • Challenges: Balancing the diverse needs of the three-way marketplace—merchants, dashers, and users.
  2. Input/Output
    • Input: User profiles, merchant data, interaction logs, and contextual signals.
    • Output: A ranked list of restaurants personalized for each user.
  3. Data: 3-Way Marketplace
    • Merchants: Restaurant profiles, menu items, pricing, location, and historical performance data.
    • Dashers: Availability, geographic locations, delivery times, and dasher ratings.
    • Users: Includes demographic data (ID, username, age, gender, city, country, timezone) and behavioral data (historical interactions with the platform).
    • User-Merchant Interaction: Click-through rates, previous orders, reviews, and ratings.
  4. Features
    • User Features:
      • Demographics: Age, gender, city, country, etc., to understand broad user segments.
      • Contextual: Device type, time of day, weather conditions, and whether the user is using the app or website.
      • Interaction Related: Historical data such as previous clicks, orders, search queries, and time spent on merchant pages.
    • Merchant Features:
      • Popularity Metrics: Average rating, number of reviews, frequency of orders.
      • Availability: Current availability for delivery or pickup, average delivery times.
    • Dasher Features:
      • Performance Metrics: Average delivery time, dasher rating, and number of completed deliveries.
  5. Candidate Generation / Retrieval
    • ElasticSearch: Utilized for fast retrieval of restaurants based on geographic and basic feature filters.
    • Approximate Nearest Neighbors (ANN) Algorithms: Employ algorithms like FAISS for more efficient and scalable retrieval based on feature similarity.
  6. First Ranking - Learning to Rank (LTR)
    • Pointwise Approaches: Treat ranking as a regression or classification problem.
    • Pairwise Approaches: Focus on the relative ordering between pairs of items.
    • Listwise Approaches: Optimize the entire list of items.
    • Deep LTR Models:
      • DCN (Deep & Cross Network): For complex feature interactions.
      • Wide and Deep: For combining deep learning with linear models.
      • DeepFM: Integration of factorization machines for feature interactions.
      • DLRM (Deep Learning Recommendation Model): Facebook’s architecture that combines dense and sparse features in recommendation models.
  7. Coarse Ranking - Personalization
    • Multi-Armed Bandit: To explore and exploit different strategies dynamically.
    • Contextual Bandit: Tailors recommendations based on the immediate context of the user.
  8. Evaluation: Online and Offline
    • Offline Evaluation: Use historical data to simulate recommendations and measure accuracy, diversity, and novelty.
    • Online Evaluation: A/B testing and multi-armed bandit testing to evaluate user interactions in real-time.
  9. Feedback Loop
    • User Feedback: Direct feedback from user interactions (clicks, orders, etc.).
    • System Feedback: Monitor system performance metrics such as latency, error rates, and throughput.
    • Adaptation and Tuning: Continuously refine models and features based on feedback to improve recommendation accuracy and user satisfaction.

Domain - NLP System Design