"Design a cron service for thousands of machines" tests whether you can reason about time, retries and exactly-once execution. Here is a compact answer outline.
1. Requirements
Users register one-off or recurring jobs (cron expressions) with a payload and a target.
Jobs should start within a few seconds of their scheduled time.
A job must not be silently lost, and should not run twice when that matters.
Scale: millions of jobs, tens of thousands firing per minute at peaks such as the top of the hour.
2. Data model
A jobs table stores the definition (id, owner, schedule, payload, retry policy). An executions table stores each run with a status: scheduled, running, succeeded, failed. The key index is on next_run_time.
3. Scheduling loop
Scheduler workers poll for jobs whose next_run_time is due, partitioned by job id hash so each worker owns a slice. A worker claims a job with a conditional update (or a lease with an expiry), pushes an execution message to a queue, and computes the next run time. Using time buckets (for example one bucket per minute) keeps the polling query cheap.
4. Execution
Executor workers consume the queue, run the job, and heartbeat while running. If heartbeats stop, the lease expires and another worker retries. Retries use exponential backoff with a max attempt count, then the run goes to a dead-letter queue for alerting.
5. Exactly-once is really at-least-once plus idempotency
Explain that a network partition can always cause a duplicate run, so jobs receive an execution id and downstream systems deduplicate on it. This is the point interviewers most often probe.
6. Hot spots and fairness
Everyone schedules jobs at 00:00, so add jitter for jobs that allow it, rate limit per tenant, and scale executors from queue depth. Mention monitoring: schedule lag (actual start minus planned start) is the main health metric.
Practice tip
Draw the claim-execute-heartbeat cycle and walk through one failure. For rehearsal I like using an AI assistant for technical interviews that throws follow-up questions at me (disclosure: I work on Linkjob AI).