Vierna & Ding
Vierna
Hey Ding, I've been thinking about how we could design a system that outperforms human players in a new turn‑based strategy game. Got any ideas on the architecture?
Ding
Sure thing. Start with a modular stack: a core engine that handles turns and state transitions, a rule‑based layer for tactical choices, and an ML layer on top that learns from game‑plays. Keep the state representation compact so the policy network can train fast, maybe using graph neural nets to capture unit positions. Use a hierarchical planner: high‑level strategy chosen by a policy, low‑level actions refined by a search tree. That way you get human‑like intuition with a performance edge. Keep the modules decoupled so you can swap in a better learning algorithm later.
Vierna
Sounds solid, but you’re still giving the ML layer too much freedom. I’d tighten the state representation to a fixed‑size vector—no more graph convolutions that keep blowing up memory. Also, the hierarchical planner needs a clear loss function; if the policy keeps choosing “human‑like” moves, you’ll stall progress. Make the high‑level strategy a discrete set of macro‑goals with hard constraints, then let the search do the heavy lifting. And don’t forget to benchmark against a pure rule‑based baseline early—any deviation should be measurable, not guesswork.
Ding
Yeah, a fixed‑size vector is cleaner, just watch out for losing spatial nuance. For the loss, mix win‑rate with a penalty for staying too close to human play. Define macro‑goals by game phase—offense, defense, expansion—and hard‑code constraints. Keep the baseline tests running so every tweak has a clear metric. If you want, I can sketch a quick pipeline to tie it all together.
Vierna
Alright, let’s outline it. Step one, define a fixed‑size state vector: encode unit counts, resource levels, and a 5×5 heatmap of threat zones, flattened. Step two, the policy network outputs one of the three macro‑goals per phase; train it with a loss = –log(win‑rate) + λ·(human‑similarity). Step three, a deterministic tree‑search (minimax with alpha‑beta) runs on the low‑level actions for 200 ms per turn, constrained by the macro‑goal. Step four, a continuous replay buffer that feeds both the policy and a value network. Keep a separate rule‑based bot for regression tests. That’s the skeleton—tweak the λ and search depth as you see fit. If you send me a skeleton, I’ll spot gaps.
Ding
Here’s a quick rough draft you can copy into a file and tweak.
StateVector = {unit_counts, resource_levels, heatmap_5x5_flat}
PolicyNetwork(inputs=StateVector) → {macro_goal: attack, defend, expand}
Loss = –log(win_rate) + λ * human_similarity
LowLevelSearch: minimax(alpha-beta) bounded to 200 ms, filtered by macro_goal constraints
ReplayBuffer: store (StateVector, macro_goal, actions, reward, next_state) continuously
ValueNetwork: predicts expected win_rate from StateVector
BaselineBot: hand‑crafted rule set for regression tests
Just plug in the exact shapes and hyper‑params, then iterate on λ and search depth. Let me know what feels off or missing.
Vierna
I like the skeleton, but it’s still missing a few crucial bits. First, the state vector needs concrete dimensions—say 20 unit types, 5 resource buckets, and the 25‑cell heatmap, so 50 numbers total. Next, decide on the policy network: two hidden layers of 128 units, ReLU, output softmax over three goals. Pick an optimizer—Adam with lr=1e-3, weight decay 1e-5, batch size 256, and run 10k steps per epoch. λ has to be tuned; start at 0.1 and watch the human‑similarity score. For the search, give the minimax a depth‑limited cutoff, not just 200 ms, and include a transposition table. Also, your replay buffer should use prioritized sampling to avoid replaying stale states. Finally, the baseline bot needs to be validated against a human‑played archive to confirm it’s actually representative. Plug those in, and we can start training.