Adversarial Co-Evolution of RL and LLM Agents in Gin Rummy

1The whole story, in one minute

Gin Rummy is a card game that needs two skills at once. You have to count fast (your "deadwood" is the cards that do not yet fit a pattern) and you have to plan ahead (you build "melds", which are runs and sets). It is a good test because it is easy to score but hard to play well, and you never get to see your opponent's cards.

Training an RL agent has a chicken-and-egg problem. We call it the opponent bottleneck. An agent is only as good as the players it practises against. Practise against a weak player and you pick up weak habits. So we built three things. A fast RL player, a perfect gold-standard opponent to grade everyone fairly, and a way to use a slow but smart LLM as a teacher. Then we tried almost every sensible way to make the RL agent stronger, and we measured each one against the perfect player.

The short version, up front. No single trick beat the perfect player. It is a very high bar. But when we stacked the ideas that really help (a better algorithm, a reward that copies the perfect player's style, a curriculum of stronger and stronger opponents, and always keeping the best checkpoint), we pushed a small agent up to 34% wins against perfect play. We also found one clear, solid result along the way. That is the next section.

2The game, and why it is hard

Each turn you draw a card and throw one away. You are trying to line up your 10 cards into melds (a run like 5, 6, 7 of hearts, or a set like three 9s). Cards that do not fit a meld are deadwood. You win by knocking (ending the hand with low deadwood) or by gin (zero deadwood, which gives a big bonus but is rare and risky). You never see the opponent's hand, so you have to play with hidden information.

3What we built: the framework

Most of the work was building a system where the RL agent, a perfect expert, and an LLM can all meet in the same game. There are five pieces.

• RL player. Action-masked PPO (and TRPO) on PettingZoo / RLCard Gin Rummy. Illegal moves are blocked, so the agent always plays a legal move.
• Gold standard. A hand-coded perfect player that uses RLCard's exact meld solver. It is the measuring stick. It never trains the RL agent.
• Distributed LLM stack. A master and worker setup so many GPUs can answer game questions at once, using the ordinary Ollama API.
• Curriculum system. It schedules opponents from random, to past versions of the agent, to strong models, so the agent always gets a fair but rising challenge.
• Web game. A no-install browser page where you can play any trained agent.

The distributed LLM server (so a 7B model can keep up with RL)

One RL training run asks the opponent tens of thousands of questions. At 0.5 to 3 seconds per call, a simple loop would take hours per round. So we keep the LLM serving separate from the training loop.

4The gold standard, and the surprise it revealed

To grade everyone fairly we built a perfect Gin Rummy player. Every turn it works out the best possible melds (exact, not learned) and knocks the moment it should. It is the benchmark. It is never a teacher.

5Everything we tried, and why each one worked or didn't

We tested almost every reasonable way to make the agent stronger, and we judged them all the same way: win-rate against the perfect player. Here is the full picture, from weakest to strongest.

6The key finding: you cannot bribe the agent into ginning

Phase 6 ran 30 controlled experiments, changing one thing at a time across algorithm, reward, and curriculum. The honest headline is that every recipe ends up near champion strength against the perfect player. But one result is clear and holds across all of them.

The numbers per recipe (each one averaged over its seeds, best checkpoint vs the perfect player):

7What moved the needle, and what didn't

Across 100+ runs, here is the honest summary of which ideas actually helped against the perfect player. Most of these are general lessons about training agents, not tricks special to this one game.

Our strongest agents (checked carefully over 2000 games each)

7bLatest experiments: does a smarter network, or search, break the ceiling?

Network architecture vs the perfect player (from scratch, relative)

Everything clusters in a narrow band (mid-20s to low-30s). Structured encoders (convolutional and permutation-invariant set / DeepSets) edge nominally above the plain MLPs, but the per-cell confidence intervals overlap — the architectures are statistically indistinguishable, and making the plain network wider or deeper barely moves it. That is strong evidence the ceiling is information-bound, not capacity-bound. These are from-scratch runs, so a relative comparison.

A search baseline (ISMCTS), graded the same way

ISMCTS stands for Information-Set Monte-Carlo Tree Search. It is not a trained network and it does no learning at all — it is a planner that thinks fresh on every single turn. The hard part of this game is that you cannot see the opponent's cards, and ISMCTS deals with that by imagining. On each turn it:

1. Guesses the hidden cards. It deals out many possible arrangements of the unseen cards that are consistent with everything it has actually observed (its own hand, what was discarded, what was picked up).
2. Plays each guess out. For every move it is considering, it quickly simulates the rest of the hand to the end, many times, across those imagined deals.
3. Picks what wins most. It plays the move that came out best on average across all those imagined worlds.

The number of "rollouts per move" is how many of those imagined playouts it runs — more rollouts means more imagined worlds explored, so stronger play and a little more thinking time per move. That is exactly the trade-off below:

These are the fair numbers: the search re-deals the hidden cards every rollout and never sees the opponent's true hand.

The surprise: played fairly, the search is weak — even at 120 rollouts it wins only about 26% vs the gold expert, below our trained agents (34%). Gin Rummy hides a lot (the opponent's whole hand plus the deck), so averaging over guessed deals is very noisy. For contrast, an oracle version that is allowed to peek at the hidden cards jumps to ~85%. That gap — 26% fair vs 85% oracle — is essentially the value of the hidden information, which is strong evidence the ceiling is information-bound: with the information you win easily; without it, even search cannot. (The in-game 🧠 Search Mastermind boss uses the oracle version so it is actually hard to play; the paper reports the fair number.)

How do our trained agents do against the search, head-to-head?

We also played each learned agent directly against the fair search. This is a like-for-like fight between a fast trained network and a slow planner:

Every trained agent is comfortably above 50% — they beat the fair search head-to-head. So at these budgets the learned agents are the stronger players; naive search is not a free win when the hidden information is large.

Is the architecture result a fluke of one recipe?

No. Re-run under different recipes (a different algorithm, a different opponent schedule), the better networks stay on top — best here is sc_ppo_asym_s0 at 30.2%. The ranking carries across recipes.

Does the method transfer to a second game?

Yes. On Leduc Hold'em, a tiny poker where the perfect strategy can be computed exactly, a simple learner graded against that perfect strategy reaches near parity (mean return -0.08 over 8 seeds; random is about −0.78). So the "grade against a fixed perfect player" idea works on a game where we can check it against the true optimum. A neural baseline (NFSP) needed far more games and was still converging (mean -0.71 over 4 seeds at the time of writing).

Live numbers refresh automatically as overnight runs land (python sweep/collect_phase8.py).

8Play the heroes yourself

A no-install web game lets you play our strongest agents, with smooth card animations. The opponent menu is a simple ladder, from a beginner-friendly bot up to the perfect player that nobody beats.

9The bottom line, and what's next

We set out to see how close a small, fast RL agent could get to perfect play, and what really makes it stronger. We built the whole framework to answer that fairly, then ran the experiments. Here is the short, honest summary.

• We pushed a small agent from the old champion's roughly 30% up to 34% against a perfect player, by stacking the ideas that really help.
• We found one clear, reusable result. The best Gin Rummy play almost never gins, and an RL agent learns the same thing on its own no matter how you reward it.
• We mapped what works (algorithm, reward style, curriculum, keep-best) and what does not (imitation, dense rewards, learned embeddings, live LLM-in-the-loop). The honest negatives are included, because they can save the next team a lot of time.

The ceiling, honestly. Tuning the reward, the algorithm, the opponents, and even the network architecture all top out around the mid-30s percent against the expert. We tested whether classic search breaks through, and it does not: a fair, no-peek Monte-Carlo search is actually weaker (about 26%), while the very same search allowed to see the hidden cards jumps to ~85%. That contrast is the key result — the ceiling is set by the hidden information, not by the method or the network. To go clearly past it you would need more information (better opponent-card inference, or vastly more search), not just more reward tuning.

10Roadmap: strengths, limits, and what is next

Here is an honest look at where the project stands today, what is strong, what is weak, and what we would do next to turn it into a paper.

What is strong

• A complete, working system end to end: a masked RL learner, a perfect benchmark to grade against, a distributed LLM server, an opponent curriculum, and a playable web game.
• A universal pipeline. The same code trains on any PettingZoo game or your own environment, not just Gin Rummy.
• A lot of careful evidence: 100+ controlled runs, all judged on one fair number (win rate vs the perfect player), with honest results for the methods that did not work.
• One clean, reusable finding: the best play almost never gins, and an RL agent learns this no matter the reward. You cannot pay it into a bad habit.
• Solid engineering: a 62× faster model-serving setup, and a self-running sweep that resubmits failures, re-collects results, and republishes this report with no human in the loop.
• Fully reproducible: every figure rebuilds from saved JSON, and training curves are on Weights and Biases.

Where it is weak (the honest part)

• We did not beat the perfect player. Our best agent wins about 34% against it. This is a clear picture of the ceiling, not a state-of-the-art win.
• The main results are on one game (Gin Rummy). The universal pipeline is checked on small games (Connect Four, Tic-Tac-Toe) with short smoke runs, not trained to convergence.
• The original idea, training against a live LLM, turned out to be too slow to be practical. The LLM ended up helping only indirectly (as a benchmark, in the design, and in an embedding experiment that also did not help).
• The methods themselves (PPO, TRPO, curriculum, keep-best) are standard. The contribution is the careful study, the benchmark, the universal pipeline, and the honest negatives, not a new algorithm.
• We did not run a search or planning method or a memory-based agent, which are the most likely ways to pass the ceiling. They are listed as future work, not done.
• Compute was limited to one cluster, and some comparison runs are short (2M steps).

What is next, in order

1. Train the universal pipeline to convergence on two or three non-Gin PettingZoo games, to show the generality with real numbers, not just smoke tests.
2. Add one search or planning baseline (counterfactual-regret or a small lookahead at decision time) and one memory baseline (a masked recurrent agent with a memory-aware evaluator), and see if either passes the 34% ceiling.
3. Test richer inputs (hand-structure features) trained to convergence. We only built and smoke-tested this.
4. Try the offline-RL path: build a large dataset of LLM-vs-LLM games and learn from it offline, so the slow LLM is used once, not inside the training loop.
5. Tighten the statistics on the headline runs (more seeds, paired confidence intervals) and turn the gold benchmark plus the curriculum harness into a small public benchmark suite.

How to reproduce

Every figure on this page is rebuilt from saved JSON results by paper/make_figures.py, and this page by paper/make_report_html.py. The sweeps run as SLURM array jobs with a watchdog that resubmits failed runs, re-collects the results, and republishes this page on its own, with no human in the loop.

# train on ANY game with the universal pipeline (coev/)
python -m coev.examples.connect_four    # any PettingZoo game, no game-specific code
python -m coev.examples.gin_rummy       # same pipeline + a gold benchmark and reward shaping

# play the web game
python game/server.py --host 127.0.0.1 --port 8000      # open http://127.0.0.1:8000

# the gold-standard benchmark
python sweep/bench_gold.py

# the final curriculum + keep-best sweeps (SLURM array + watchdog)
python sweep/curriculum_configs.py && sbatch --array=0-29%10 slurm/curriculum.slurm
python sweep/phase7_configs.py    && sbatch --array=0-8%6 --export=ALL,CFG_DIR=phase7_cfgs slurm/curriculum.slurm

# regenerate this report
python paper/make_figures.py && python paper/make_report_html.py

The typeset paper is the arXiv version. Live training curves are on Weights & Biases (groups phase6-curriculum, phase7-ceiling).