You say that, but that also definitely both runs counter to my experience, and runs counter to (e.g.) jailbreaking models.
I don't doubt the model gets the full context. I have doubts that it's able to execute on it in a way that would be deemed successful for moderately long conversations
The "breaks down on long conversations" problem has two vectors.
The first vector is the prompt being compacted/dropping off. The solution: ensure the prompt is preserved fresh and complete on every input. Very easy to do.
The second vector is if the size of the historical context is so great that the model starts to fail to weigh the prompt and the latest message correctly. This is the part where the model breaks down entirely - not just in terms of failing to keep the secret prompt in mind, but also in terms of just outright failing to respond to what the latest input actually says because all of its attention is on history.
The second vector has two solutions. Model training is the first one. Newer, larger, and particularly closed frontier models tend to have leaps and strides in training specifically to rigidly adhere to system prompt and to stay on task for the latest user input, that's just a huge part of the underlying training. So you make sure your model is up to the task.
The second is that you benchmark the total context window/ratio of history to prompt+message where things start to break down.. and you limit the context window before that and keep compacting history more and more, progressively, as it approaches that limit. Yes, it will forget details about interactions long ago. You will too; that's a very human thing. The trick, like I said, is saving the actual important persistent changes via game flags affecting the prompt, which is never compacted or left off. Exactly like a human: long term recollection (important stuff committed to memory) is good for important details, short term memory is pretty good for acting on what just happened, and recent memory slowly degrades until you can't remember what you had for lunch two weeks ago.
It's 100% possible. The reason we haven't seen it yet is because it's still financially inadvisable for a game developer to actually try it... But as we can see, it's starting.
Sure, I can believe both. Which does handle creating contradictions over long conversations. _Butttttt_ I really am still pretty unconfident that it'll create a particularly great experience. I am deeply aware human writers can be pretty terrible, but my experience with LLMs is that I can expect them to be better than a person off the streets, but I have generally never found them to be exceptional at things. They're worse than what I can do in my fields of speciality, but better than what I can do in arbitrary other fields.
I think they're going to appeal to different kinds of players, personally. Some are going to feel like the character in the game is really responding to them, personally, and that customised experience is going to be more meaningful to them than one that is more polished.
Others are going to respond more positively to outright better, human, creatively written dialogue.
When it comes to AAA games especially we're not really comparing this to your average Joe writing dialogue, we're comparing this to industry professional writers who are usually very good at what they do. They're going to gap any kind of model efficient enough to do this kind of thing at scale.
But, yeah. Some people are definitely going to not even notice that gap if they're just talking to a character and that character is just responding to them in a personal way.
As for me... I'm not entirely unskeptical, I just believe it can absolutely technologically be done with the tech we have today, it's more down to how it's done and whether it makes any financial sense to do.
0
u/ShellTrajectory 1d ago
You say that, but that also definitely both runs counter to my experience, and runs counter to (e.g.) jailbreaking models.
I don't doubt the model gets the full context. I have doubts that it's able to execute on it in a way that would be deemed successful for moderately long conversations