Choosing parallel workers per stage

Super Speedy Imports runs four stages in parallel by default — load-csv, attach-existing-images, upload-remote-images, fix-attributes. Each spawns N worker processes that chew through a slice of the data simultaneously. The default N is 8.

8 workers is the right answer for a server with plenty of RAM and CPU. On a smaller box, 8 will swap-thrash MySQL and cost you wall-clock time instead of saving it. This article covers how to pick the right number.

Where to set it

SSI Admin → Performance Options in the WP dashboard. Two scopes:

• Default workers — applies to every parallel stage on every import.
• Per-stage workers — overrides the default for specific stages (e.g. 8 workers for load-csv but 2 for upload-remote-images because the latter is network-bound).

Per-stage override is in the table below the default — pick the stage from the dropdown, set the worker count.

The CLI also accepts a --workers=N flag that overrides everything for the duration of one run:

wp ssi 5 --workers=4

Useful for one-off tuning without changing the saved Performance Options.

How parallel work is split

Each parallel stage decides how to slice the work:

• load-csv — splits the CSV file by byte offset. Each worker reads a contiguous range of rows. Splitting is precomputed so workers never overlap.
• attach-existing-images and fix-attributes — split the batch table by post_id range. Each worker owns a disjoint set of post IDs.
• upload-remote-images — assigns each unique image URL to exactly one worker (the worker holding the lowest batch.id referencing that URL). Guarantees no duplicate uploads even when many products share an image.

The slicing math means N workers always handle exactly total_rows / N items each (give or take a few). No worker steals from another mid-run.

What workers cost you

Each worker is a separate PHP process. Cost per worker:

• ~150-300MB RAM (PHP heap + WP bootstrap + plugin code).
• One MySQL connection (held for the duration of the worker’s stage).
• One file handle on the CSV (for load-csv) or per-image (for image stages).

At 8 workers that’s 1.2-2.4GB of PHP-side RAM, 8 MySQL connections, and 8 file handles. Modest for a dedicated server, painful on shared hosting.

MySQL also gets hit harder with more workers — each contributes INSERTs / SELECTs / UPDATEs to the same tables. With low max_connections (default 151), 8 SSI workers + your web traffic + other plugins can saturate the pool.

How to size it

A practical decision tree:

Step 1 — check total RAM

free -h

Total RAM	Recommended worker count
< 2GB	Don’t run SSI here. Upgrade the host.
2-4GB	2 workers. Anything more swap-thrashes.
4-8GB	4 workers.
8-16GB	6-8 workers.
> 16GB	8 workers is the default; rarely useful to go higher.

These are conservative numbers that assume MySQL also needs RAM on the same box. If MySQL runs on a separate server, you can push the PHP-side worker count higher.

Step 2 — check MySQL max_connections

wp db query "SHOW GLOBAL VARIABLES LIKE 'max_connections';"

Default is 151. SSI workers want 1 connection each. Your web traffic also wants connections (typically 50-100 concurrent for a small WP site). If max_connections is at default and you’ve configured 8 SSI workers PLUS expect web traffic to land during the import, raise max_connections to at least 200 or schedule the import for off-hours.

See Database tuning for the full DB sizing recipe.

Step 3 — verify with a small run

The 2.49+ CLI prints per-stage timing at the end of every run:

load-csv               Rows Processed    868,000   1098.28s

Re-run with a different worker count (--workers=4 for a one-off) and compare wall-clock for the same stages. If 4 workers ran in 80% of the wall-clock that 8 workers took, you’re either I/O-bound (more workers don’t help) or memory-pressured (more workers actually hurt because of swap).

If 4 workers took 200% of 8-worker wall-clock, the bottleneck is CPU and you should keep 8.

When fewer workers is faster

Counter-intuitively, dropping worker count can speed things up:

• MySQL is swap-thrashing. When MariaDB’s working set exceeds available RAM, the buffer pool spills to swap. Adding workers makes this worse — each one issues more queries against a database that’s now bottlenecked on disk I/O. Reduce workers; cap innodb_buffer_pool_size to fit in RAM.
• Image stages hit a slow remote server. upload-remote-images workers wait for HTTP responses. Eight workers hammering the same vendor CDN at 200ms/request means 1600ms of total wait time across the eight requests, NOT 200ms. If the upstream rate-limits at, say, 10 requests/second, 2 workers is the same speed as 8 and uses less local RAM.
• You’re on shared hosting with a CPU quota. Many shared hosts kill processes that sustain high CPU. Eight PHP processes at 80% CPU each can trip the quota. Two workers sustained 60% each can fly under the radar.

When more workers won’t help

Three stages remain SEQUENTIAL no matter what:

• import-taxonomies — as of 2.55.8 this stage is forced to a single worker by ssi_perf_effective_workers() (functions/perf-options.php). Parallelism measured only ~7% faster at 100k (1418s single vs 1326s with 8 workers — near-zero scaling) because the stage is dominated by row-lock contention and the single-threaded recursive-CTE hierarchy finalize, not per-row CPU. Term creation now also recovers from a duplicate-key race via term_exists() instead of aborting, so the old wp_terms UNIQUE collision is no longer a reason to parallelise.

Heads-up: the Performance Options screen still renders a Workers dropdown for Import Taxonomies, but it is silently ignored — only the “Hierarchy batch flush size” (hierarchy_batch_size) knob has any effect on this stage. Don’t expect raising its worker count to do anything. – upsert-relationships — a single bulk INSERT ... SELECT DISTINCT does the whole stage; splitting it doesn’t help (it’s bottlenecked on MySQL, not PHP). – update-posts, insert-posts, update-postmeta, insert-postmeta — same story; single bulk SQL statements.

For these stages, the right tuning lever is database tuning (innodb_buffer_pool_size, the enable_db_tuning opt-in session bumps) — not worker count.

Common configurations

Daily price-only refresh on a small (4GB) box

• Default workers: 2
• Import mode preset: Update prices / postmeta only
• enable_db_tuning: off (the only sequential stage that benefits is upsert-relationships, which this preset skips)

Monthly full catalog refresh on a medium (8GB) box

• Default workers: 4
• Import mode preset: Full import
• enable_db_tuning: on (the long sequential stages benefit)

Big-bang 1M-row migration on a dedicated (16GB+) box

• Default workers: 8
• Per-stage override: upload-remote-images → 4 (if the image source is rate-limited)
• Import mode preset: Full import
• enable_db_tuning: on

CI / test environment

• Default workers: 1 (forces the single-worker bypass code path — exercises the same code path real users hit on tiny imports without paying the proc_open overhead)

What’s next

• Database tuning — server-side sizing that affects how many workers you can sustain.
• Diagnosing slow imports — what the per-stage timing tells you about where to focus tuning.
• How imports work — the stage pipeline — which stages are parallel vs sequential and why.