- What is the difference between training and inference?
- What is a token?
- Why does a longer prompt usually cost more?
- What is time to first token?
- Why can an LLM return different answers to the same prompt?
- What are prefill and decode?
- Why does KV cache use GPU memory?
- What metrics would you track for an LLM service?
- How would you benchmark a model server?
- What are the risks of quantization?
- How would you debug high p99 latency in an inference service?
- What autoscaling signals are useful for LLM serving?
- How would you roll out a new model safely?
- How would you reduce inference cost without hurting quality?
- What should be logged, and what should not be logged, for privacy?
Design a chat service that uses an LLM to answer questions from company documents.
Explain:
- How documents are retrieved.
- How prompts are built.
- Which metrics are collected.
- How output is validated.
- How the system scales.
- How cost is controlled.