magento2 – Load times for files in pub/static/ are extremely variable and often painfully slow

On Magento 2.3.3 I’m seeing extremely variable load times for files served from Magento’s pub/static directory. This happens in both developer and production modes, and I can’t understand what’s causing it.

Main symptoms are pages which take a huge amount of time to load, especially on first visit and especially when the FPC is disabled. But I don’t just mean “a few seconds extra” – I’m talking minutes of load time.

pub/static resources (mainly JS files) take several seconds each to reach TTFB i.e. there’s tons of “waiting” for files in the browser. This causes a massive amount of request blocking as the browser waits.

Even if I load up the URI of a single JS file in the browser the load times are insanely variable: first time I request a file directly it might take 4 seconds to load, then I refresh a couple of times and it loads instantly, then I refresh again and it’s back to 4-5 seconds. And this is on a machine which has zero net traffic other than my own testing.

The hardware I’m using is a 32-core CentOS-based machine with 128GB RAM and SSDs so this should not be a hardware issue. I’m running httpd, MariaDB, php-fpm, redis/memcache for page cache and session storage. I’ve set up Varnish in the past but would prefer to get Magento running acceptably before resorting to using aggressive caching as a crutch.

My php-fpm config looks like this:

listen = /var/run/php-fpm/website.sock
listen.owner = magento = www
listen.mode = 0660
user = magento
group = www
pm = static
pm.max_children = 500
pm.start_servers = 30
pm.min_spare_servers = 30
pm.max_spare_servers = 50
pm.max_requests = 500
php_admin_flag(log_errors) = on
php_admin_flag(zlib.output_compression) = On
php_value(memory_limit) = "4096M"
php_value(session.gc_maxlifetime) = "86400"
php_value(opcache.memory_consumption) = 512MB
php_value(opcache.max_accelerated_files) = 60000
php_value(opcache.consistency_checks) = 0
php_value(opcache.revalidate_freq) = 0
php_value(opcache.enable_cli) = 1
php_value(opcache.interned_strings_buffer) = 64

Any ideas?