Project1998 · server performance

Load run 1 — where the server stalls

The first bot load test against the server. Twenty-five clients up to four hundred, on one map, on the laptop. The server holds to two hundred players without missing a tick; the first stall is at four hundred, and below that the thing that gives out is the test client, not the server.

2026-09-12 · server master b5bb486 (Debug) · test client main 101b679 · loopback

First server stall

400 players

clean at 25, 50, 100, 200

Tick work at 400

132 ms

median, of a 333 ms period

Lock-wait

0 ms

in all 26 slow ticks

Server CPU, percent of one core — the machine has sixteen

The 200 and 400 figures come from the runs with the client's event stream off — the only way those counts were reachable. 400 is measured over the hold window.

Slow ticks recorded in the log window

A zero means the log window was read and held no slow tick, not that it went unchecked. Slow sends follow the same shape: none below 400, then 58,208.

What actually failed

Below four hundred players the limit was the test client. The run below is the same script against the same server, differing only in whether the client was serialising every frame to its JSON event stream.

Players	Client event stream on	Client event stream off	Server said
25	passed	not run	nothing
50	passed	not run	nothing
100	1 of 2 aborted	not run	nothing
200	client aborted	passed	nothing
400	not run	passed	26 slow ticks

With the stream on, the client aborts at two hundred players because bots stop seeing their own chat echo inside its five-second deadline. With it off, the same script and the same server carry two hundred players in ninety seconds and four hundred in a hundred and seventy-one. At one hundred the client sits right on that deadline — one of the two runs passed and one aborted — while the server showed nothing wrong in either.

The stall at four hundred

SLOW TICK: work 297ms (lock-wait 0ms), late 25ms, gc 62ms — 400 player(s), 305 mob(s) on 1 active map(s)

Twenty-six slow ticks, work at 83 ms minimum, 132 ms median and 297 ms worst against a 333 ms heartbeat and an 83 ms threshold. The lock-wait figure was zero in every one of them and garbage collection never exceeded 62 ms. Read the way the server's own instrumentation asks to be read, that means the tick body is too big for the population it is driving — not lock contention, not GC, not the OS failing to schedule the thread. Outbound queues reached 326 frames per client at the median. Server CPU over the hold was 61% of one core out of sixteen, and the working set peaked at 713 MB.

The wire dump is not behind its own switch

Turning the game's wire dump off (P1998_LOG_WIRE=0) saved 14–31% of the log volume and no measurable CPU. That is less than expected, and the reason is that the outbound half of the dump is not actually behind that switch: SendMap in Server/Session.Dialog.cs logs a hex dump of every packet it sends regardless, while the inbound side in Server/Session.cs is properly guarded. At four hundred players the logger dropped 663,928 lines and rolled a 64 MB file twice in under three minutes. It is a one-line fix.

What these numbers do not cover

Every server figure is a Debug build — the launcher runs --no-build against bin/Debug. Treat four hundred players as a floor, not a ceiling.
Every bot spawns on the same tile of a 16×16 map, and the server refuses a step onto an occupied tile, so 97.5% of walks at twenty-five players and 100% at fifty and above were refused. Chat broadcast was exercised hard; movement broadcast essentially was not. The next run needs a script that spreads the bots out.
Three of the twelve runs had their server log rotated away before it could be read. Those are reported as unread, never as clean.
No GM account was on the bot roster, so nothing was fought. The optional combat block stayed off.
Nothing on the server was changed for this run.

What follows from it

Gate the outbound packet hex behind the wire switch — one if.
Cut the tick body at high population on one map. This is the one standing between the current server and hundreds of concurrent players.
Bound and measure the per-session outbound queue under chat fan-out: say from N players on one map is N² frames, and at four hundred that is 160,000 a cycle.
A thread-pool floor sized to the connection count.
For the test client: make the event stream affordable at scale, put per-bot timings in the checkpoint file so the numbers survive turning it off, and make the five-second verb deadline a measurement rather than an abort.