# SSI vs WPAI benchmark — runner + data

This folder contains the benchmark runner script, the per-run JSON results, sar telemetry logs, product export CSVs, and the manual command reference used to compare Super Speedy Imports against WP All Import Pro at 1k and 50k product sizes (with and without images, fresh and update).

## What you'll find here

| File | Purpose |
|---|---|
| `benchmark-runner.py` | The Python orchestrator. Restores snapshot, starts sar telemetry, starts HTTP probes, runs the import, collects everything, exports results, saves JSON |
| `MANUAL-COMMANDS.md` | Step-by-step manual SSH commands for the runs the orchestrator couldn't handle (multi-day WPAI) |
| `README.md` | This file |
| `<run-name>.json` | Per-run results (timings, probes, product counts, parsed import metrics) |
| `sar-<run-name>.log` | Raw `sar -u -r -d 5` output (CPU, RAM, disk I/O at 5-second intervals) |
| `export-<run-name>.csv` | Full product export via `wp ssi export` — every product's normalised data, used for SSI vs WPAI field-by-field diffing |
| `benchmark-run.log` | Stdout from the runner across all runs (start/finish markers, errors) |

## Prerequisites for running the script yourself

### Server side

- A test box reachable via `ssh benchmarking` (SSH alias for your server, root login or sudo)
- WP-CLI user `tests46427` (or change `WP_PATH` / user in the script)
- WordPress site at `/var/www/tests.superspeedyplugins.com/htdocs/` (or change `WP_PATH`)
- WooCommerce, Super Speedy Imports, WP All Import Pro (+ WooCommerce Add-on), and the test admin user installed
- Hetzner Cloud account with the test server in a project + an API token
- A Hetzner snapshot of the clean baseline (server with WP+Woo+SSI+WPAI installed, **0 products**, both SSI import and WPAI import configs created and pointing at the same CSV file)

### Local side

- Python 3.x with stdlib (no extra deps)
- `ssh` and `scp` working against `benchmarking`
- `curl` for the frontend probes (probes go through Cloudflare from your machine — wp-admin probes run server-side via SSH to bypass Cloudflare)

### Config inside `benchmark-runner.py`

Edit the top of the script:

```python
SNAPSHOT_ID = 390805640          # your "benchmark-ready-empty" snapshot
HETZNER_TOKEN = "..."             # your Hetzner project API token
SERVER_ID = 132884991             # your server's numeric ID
WP_PATH = "/var/www/.../htdocs/"  # WP root on the test server
SSH_HOST = "benchmarking"         # SSH alias
DATA_DIR = "/var/www/.../super-speedy-imports/.benchmark-data"  # where to save results
SITE_URL = "https://your-test-site.example.com"  # for HTTP probes
```

### Helper PHP files

The runner expects two helper scripts in `/tmp/` on your local machine (it SCPs them to the test server during restore). Create them:

```bash
cat > /tmp/count-products.php << 'EOF'
<?php
require_once("/var/www/tests.superspeedyplugins.com/htdocs/wp-load.php");
echo wp_count_posts("product")->publish;
EOF

cat > /tmp/count-products-all.php << 'EOF'
<?php
require_once("/var/www/tests.superspeedyplugins.com/htdocs/wp-load.php");
$c = wp_count_posts("product");
foreach ($c as $s => $n) { if ($n > 0) echo "$s\t$n\n"; }
EOF
```

(Edit the path inside each PHP file to match your WP root.)

### A wp-admin user for the probes

The runner logs into wp-admin from the server to measure admin page response times. Create the user once:

```bash
ssh benchmarking "sudo -u tests46427 wp user create benchmark benchmark@test.local \
  --role=administrator --user_pass=BenchmarkAdmin2026! \
  --path=/var/www/tests.superspeedyplugins.com/htdocs/"
```

If you change the password, update the `login_wp_admin()` function in the runner.

## Running the orchestrator

Once the prerequisites are in place:

```bash
cd /var/www/superspeedy/htdocs/wp-content/plugins/super-speedy-imports/.benchmark-data/
python3 benchmark-runner.py 2>&1 | tee benchmark-run.log
```

The runner will:

1. **Restore the snapshot** (10 sec via API, 60-120 sec for SSH to come back)
2. **Login to wp-admin** (server-side to bypass Cloudflare) and stash a cookie
3. **Start sar** on the server (background process writing to `/tmp/sar-<run-name>.log`)
4. **Start the probe collector thread** locally — hits homepage, /shop/, /product-category/men/, /product-category/women/, /cart/, /wp-admin/edit.php?post_type=product, /wp-admin/ every N seconds
5. **Run the import** (SSI or WPAI as configured)
6. **Stop telemetry**, scp sar log back to local
7. **Export products** via `wp ssi export` (works for both plugins' results)
8. **Save the JSON** to `.benchmark-data/<run-name>.json`
9. **Move to the next run**

### Configuring which runs to execute

Edit `run_all()` at the bottom of `benchmark-runner.py`. The default sequence is:

```python
# Run 1: SSI 1k fresh
run_benchmark("ssi-1k-fresh", "ssi", 1, restore=True, probe_interval=5)
# Run 2: WPAI 1k fresh
run_benchmark("wpai-1k-fresh", "wpai", 1, restore=True, probe_interval=10)
# Run 3: SSI 50k fresh+images
run_benchmark("ssi-50k-fresh-images", "ssi", 2, restore=True, probe_interval=30)
# Run 4: SSI 50k update (NO restore — runs against Run 3's data)
update_csv = f"{SSI_UPLOADS}/shirt-test-50k-update.csv"
run_benchmark("ssi-50k-update", "ssi", 2, restore=False,
              file_override=update_csv, probe_interval=30)
# Run 5: WPAI 50k fresh+images
run_benchmark("wpai-50k-fresh-images", "wpai", 2, restore=True, probe_interval=60)
# Run 6: WPAI 50k update
run_benchmark("wpai-50k-update", "wpai", 2, restore=False,
              wpai_file_overwrite=f"{SSI_UPLOADS}/shirt-test-50k-update.csv",
              probe_interval=60)
```

Comment out runs you don't want, or duplicate runs to repeat them.

### Function arguments

- `run_benchmark(run_name, plugin, import_id, restore, file_override, probe_interval, wpai_file_overwrite)`
  - `run_name`: identifier for the output files (e.g. `ssi-50k-fresh-images`)
  - `plugin`: `"ssi"` or `"wpai"`
  - `import_id`: the SSI or WPAI import ID to run
  - `restore=True/False`: whether to restore the snapshot before running. `False` for update runs that depend on a previously populated DB
  - `file_override="/path/to.csv"`: for SSI updates, runs `wp ssi <id> --file=<csv>` to use a different CSV
  - `probe_interval=N`: seconds between HTTP probes. Use 5 for short imports, 60+ for multi-hour runs to keep probe data file size sane
  - `wpai_file_overwrite="/path/to.csv"`: for WPAI updates, copies the update CSV over the original before launching WPAI (WPAI doesn't have a `--file=` flag)

## ⚠ KNOWN ISSUE: multi-day imports break the SSH connection

`benchmark-runner.py` uses a single long-running SSH command to run the import. For SSI (3 hours) this is fine. For WPAI on a slow box doing 50k with images (66 hours) the SSH connection drops and the runner crashes — but the import keeps running on the server.

**If the runner crashes mid-run** you'll lose the in-memory probe samples (the JSON write happens at the end of each run, not incrementally). The sar log on the server survives because it's a separate background process.

**Workarounds for multi-day runs:**

1. **Manual approach (as used for runs 5-7)**: see `MANUAL-COMMANDS.md` — launch the import via `nohup ssh ... wp all-import run 2 ... &` and poll separately. You lose HTTP probe data but keep sar + final timing + exports.

2. **Future fix idea**: change the runner to launch the import as a background process on the server (`nohup ... &`) and poll for completion + write probe samples to disk incrementally rather than holding them in memory. Then a runner crash only loses the most recent few samples, not all of them.

If you re-run this benchmark and want to keep probe data for the 50k WPAI run, **fix the runner first** along those lines.

## Reading the data

```python
import json
d = json.load(open("ssi-50k-update.json"))
print(f"Wall clock: {d['wall_clock_s']}s")
print(f"Items per second: {d['items_per_second']}")
print(f"Products before: {d['products_before']}, after: {d['products_after']}")
print(f"Probe samples: {len(d['probe_samples'])}")
print(f"SSI stage metrics: {d['parsed_metrics']}")
```

The probe samples are an array of dicts like:
```json
{
  "homepage":      {"status": 200, "time_s": 0.339},
  "shop":          {"status": 200, "time_s": 0.215},
  "category_men":  {"status": 200, "time_s": 0.221},
  "category_women":{"status": 200, "time_s": 0.224},
  "cart":          {"status": 200, "time_s": 0.223},
  "admin_products":{"status": 200, "time_s": 0.295},
  "admin_dashboard":{"status": 200, "time_s": 0.281},
  "category_men_product_count": 0,
  "category_women_product_count": 0,
  "timestamp": "2026-05-29T20:27:01.234567",
  "elapsed_s": 17.0
}
```

The sar logs are raw text from `sar -u -r -d 5`. Use `parse_sar_file()` from the runner to convert them to time-series JSON.

## Reproducing the CSV files

The shirt-test CSVs are generated by the Rust generator in `super-speedy-imports/create-test-csv/`:

```bash
cd /var/www/superspeedy/htdocs/create-test-csv/
./create-large-clothing-test-csv 1000   # produces shirt-test-1000.csv
./create-large-clothing-test-csv 50000  # produces shirt-test-50000.csv
```

The "with images" variant references images from `wp-content/uploads/benchmark-images/img-NNNNN-{small,large}.jpg` — generate these once and bake into the baseline snapshot. The "-update" variant is generated by `make_update_csv.py` (see the conversation log around 2026-05-27 for the script — basically: keep 95% of rows, modify 5%, remove 2.5%, add 2.5%).

## Why the dot-folder?

`.benchmark-data/` is a dot-folder so:
- Git ignores it by default in most build pipelines (won't ship to production)
- BUT can be force-added (`git add -f .benchmark-data/`) to source-control the runner + the results
- Survives snapshot restores (it's local to your machine, not the test server)
- The user requested this explicitly after losing data to `/tmp/` getting wiped during snapshot restores
