One VM per AI inference instance on a 48-core, 96-thread Xeon Platinum 8558U — when traffic spikes, does everyone slow down together?

  They will, unless isolation is built first. Ninety-six threads split across dozens of light VMs leaves each tenant comfortable at rest; the 4GHz boost keeps single-instance replies respectable, and 260MB of L3 trims memory round-trips when instances share one small model. When traffic peaks they contend for the same memory bandwidth, and without weighted scheduling, everyone slows in lockstep.

  Teams running dozens of light AI services, none of which saturates a socket alone, fit this 48-core base well; a single instance meant to own the machine is better served bare-metal. Pin down per-VM vCPU and memory quotas before promising concurrency — otherwise the 8558U's core count only makes the jam more spectacular.