Teaching a Robot how to Game with Turnstone and Nvidia VSS!

TL;DR

Four Linux machines automatically play and capture games using Turnstone AI orchestration and Nvidia VSS to build an automated hardware/driver bug-detection test harness.

Key Points

• 1.Four parallel Linux gaming machines form the test bed. Configs include a GeForce GPU, Strix Halo GMK Tech Mini PC, AM5 7600X system, and a Titan X system, each with a CYP Nano KVM capturing 1080p60 video over the network.
• 2.Turnstone is the deterministic AI orchestration layer. Unlike freeform agents, Turnstone builds repeatable scaffolding where AI assists in creating processes but humans remain in the loop for creative or ambiguous decisions.
• 3.Nvidia VSS3 (Video Search and Summarization) analyzes game footage without fine-tuning. Out of the box it recognized Shadow of the Tomb Raider gameplay, detected a stealth kill (+30 XP), flagged lighting transitions, and scored clips 1–10 for human review priority.
• 4.The platform can catch subtle visual bugs traditional automated testing misses. Examples include desktop flickering on Bazzite from a driver bug and black screen cutouts caused by a faulty HDMI cable — both identified by the AI reviewing captured clips.
• 5.Running VSS locally requires serious hardware. The setup uses four RTX Pro 6000 GPUs (minimum two recommended), and even that struggled under multiple simultaneous streams due to unoptimized configuration.
• 6.AI tooling compressed a 150-hour dev project to roughly 15–20 hours. The real excitement isn't AI playing games well — the Nitrogen model only processed ~1 FPS — but AI-assisted scaffolding that makes automated end-to-end Linux driver testing newly feasible and affordable.