> The Jeopardy! match with Watson was essentially rigged due to the computer’s superhuman reaction time.
From many anecdotes I've heard from previous Jeopardy contestants, buzzer technique is more than half the battle. That's part of the reason Ken Jennings was able to go on winning as long as he did - he continuously refined and developed better intuition for when to buzz.
Many contestants (a large portion of whom likely participated in college Trivia Bowls and were seasoned trivia buffs) could answer the vast majority of Jeopardy clues with relative ease.
The reason for this is that Jeopardy, in order to have household appeal, cannot legitimately engage in an increasingly difficult arms race for its questions - it has to have a capped ceiling in terms of clue difficulty.
> Recently, however, that long-held truism has been shaken: The show has started releasing statistics about buzzer attempts, revealing that some contestants, especially those in the game’s upper echelons, ring in much more often than others. It’s not that the buzzer doesn’t matter—it just matters much less than was previously believed, a shake-up that has reelevated the importance of subject-matter expertise.
Difficult questions are fun, there's a reason the UK show "Only Connect" does well. Yeah, I'm not going to know what's going on when the question relies on knowing the difference between Finnegans Wake and Finnegan's Wake, but when you list decreasingly obscure eliminated Google products I know what's going on by clue three even if many contestants only even recognize the fourth clue.
Among HN readers I'm guessing "Control" and "Shift" already set people on the correct path before they see "Alt" or "Option" but a lot of word focused people doing such a quiz will be studying the words - counting vowels, thinking about meanings in other languages before they realise it's about keyboards.
Jeopardy already puts a small spin on things with the reverse format, they could work on that harder to deliver a more challenging quiz that's still getting the audience.
I really like NAQT's (National Academic Quiz Tournaments ) take on questions, where it goes from more obscure to less obscure the further you get into the text. For example:
Though he suffered from Bright’s disease, he allowed himself to be considered for his party’s nomination, but he lost to James G. (*) Blaine. Once known as a Stalwart, he declared his independence by signing the popular Pendleton Act. For ten points—name this New York politician who acceded to the presidency upon the 1881 assassination of James Garfield.
The asterisk indicates bonus points if a player correctly answers the question before the host finishes reading past that mark.
But this kind of format where you can "buzz in early" would be a pretty drastic departure for the show.
There were two astounding takeaways when Watson beat Ken and Brad:
1. We've advanced the state of the art in terms of being able to reason and parse complex language to a level the general population wasn't aware of.
2. Beating humans on the buzzer, of all things, feels intrinsically unfair and there's no graceful way to account for that.
Watson's ability to interpret, correctly, what most of the Jeopardy! "answers" were looking for was a sea change. It really was as big as Deep Blue, the crossing of a frontier we collectively thought would likely be much further in the future, and it was here now: complex clues, difficult to reason out context, and the ability to search its corpus of knowledge and assign a confidence value and then produce accurate results... it was mind blowing.
It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now, owing to the specific nature of generative transformers and the advent of attention, specifically. But Watson was pre "attention is all you need!" It was blood, sweat, and tears, and markov chains, and all kinds of craziness. It was a huge swing, and the fact that it actually worked was remarkable.
But back to the buzzer: most people sitting at home probably don't realize that the buzzer is locked out from buzzing in until the host (then Alex, now Ken) completes the read of the clue. At that point, an off-stage producer presses a switch to unlock the buzzers. So prospective players spend their weeks before their tape date spamming a retractable pen in front of the TV trying to nail the timing from the cadence of the host's voice, and hoping they can get it right. Because getting it wrong penalizes you: if you attempt to buzz in before the lock is lifted, your buzzer is individually locked for another 250ms or so. Another producer has a screen that monitors the buzzer activity so they can see if players are habitually buzzing in too early and getting locked, or to monitor for malfunctions, and it's wild to watch it light up like a Christmas tree in the 100 or so last milliseconds of the clue.
This is why you see players spam the buzzer: if they're locked out, the best thing they can do is keep attempting to get back in until the lock lifts. This is also why on rare occasions, you'll see all three players spamming and no one actually getting buzzed in: everyone locked.
So how do you translate this to a computer? The answer was basically to create a makeshift "finger" that would activate the buzzer, but would consistently buzz in at exactly 28ms after the lock cleared... every single time (so long as its confidence was high enough).
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer. It's a signal loop: wait for lock to clear, receive signal lock has cleared, activate buzzer 28ms later. Every device with a stopwatch or timer or clock is more accurate than a human attempting the same timing exercise. It's a solved problem, and not an interesting one. And it disproportionately took the wind out of the sails from the human competitors, who are both the most world-class, best-on-the-planet Jeopardy! players, who were losing by milliseconds to that signal loop.
It's still remarkable Watson was able to successfully answer so many clues correctly and end up winning, but the buzzer aspect was an (intractable) bummer.
If I recall, Watson was fed the whole text of the answer as soon as the other contestants could see it. Personally, I thought it would be fairer to have it do speech-to-text and/or OCR to level the playing field.
But I suppose these constraints are just targeting the machine's input mechanism and not its actual reasoning ability once the answer is read. I'm curious how Watson at the time could handle a particular category that Ken dominated: "Initials to Roman Numerals to Numbers" [1]
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer.
As anyone who has experienced early video game consoles can attest --- even the <2MHz CPU of a NES is far more than fast enough to beat a human at reaction time, which is what makes it all the more impressive when humans do things like https://news.ycombinator.com/item?id=38841080
>It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now
“Answering Jeopardy questions” is one of those tasks where being close to correct is not good enough. A system that hallucinates plausible-sounding answers would be a kind of awful approach to creating software that’s meant to 100% a game like Jeopardy
But a system with a perfect memory and sufficient search space would be great at it. This is precisely where the stochastic database, even a huge Markov model would work great.
Only the hard wordplays maybe would pose a problem, and statistically you can only go so hard on these before human players start lagging too.
Remembering a ton of facts is something the computer would be excellent at...
I mean, it's not "close to correct". It's... really almost 100% correct.
Go try it. Hit up J-Archive, prompt it that you'll feed it Jeopardy answers and want a simple response in the form of a question. I've yet to stump it, even with stuff like Rhyme Time: "a racket-&-net sport peril" (tennis menace) or other more complex clues. Every once in awhile, it'll switch to the thinking model, but most were immediate and correct.
I'm really curious how this kind of writing will be perceived in the coming years. It's "good" long form writing but a lot of that also feels like journalistic padding for the sake of the form, as if you asked an LLM to "write in the style of a long form journalist."
Easy example:
"...dismissed as “disappointing,” “unfair,” and—my favorite—“a gimmick”..."
Is it really important for me to know Clare McNear's favourite? A couple of years ago I wouldn't have thought anything of it but now I'm so sickened constantly by LLMs adding so much needless cruft to everything they write I'm reading this thinking "just tell me the story, for the love of humanity!"
I’m seeing hyperreality in real time. You complain that an article of long form journalism appears to be the sort of thing that might be produced by asking an LLM to create it?
Doesn’t that mean it is true to form? Can you not imagine that some people like reading stories written by others?
Yes, that's exactly what it means. Like I say, a few years ago I wouldn't have thought anything of it, but now it's behind painfully clear how formulaic that kind of writing is.
> Can you not imagine that some people like reading stories written by others?
This is quite uncharitable and a straw man. I love reading stories written by others. I'm complaining very specifically about the over reliance on tropes of the form.
I also wasn't leading with the complaint, I was leading with the curiosity I feel about how writing will develop as a result of what I'm seeing here, as I'm sure I'm not the only one who suddenly feels different about human made writing. That's not too say I don't lament it, either, but it is what it is.
I think it's valid to say that the advent of LLMs has diminished the value of formulaic long-winded prose, and increased the value of being brief and concise.
Human long-form writing will be perceived as the top of value.
What is more valued today, IKEA furniture or hand-made natural wood furniture? A mass-produced Timex quartz watch or a hand-made mechanical watch? A painting or an ink-jet print of a painting?
Good, creative, original human long-form writing, yes, I'm sure.
But that's quite different from something essentially indistinguishable from AI output. The article here is very by-the-numbers and doesn't really show any strengths of the human mind over the ability of LLMs, does it? Or can you tell me why you enjoyed hearing which was her favourite quote, and why that makes the article akin to a handmade mechanical watch?
https://archive.is/p2hG5
From the article:
> The Jeopardy! match with Watson was essentially rigged due to the computer’s superhuman reaction time.
From many anecdotes I've heard from previous Jeopardy contestants, buzzer technique is more than half the battle. That's part of the reason Ken Jennings was able to go on winning as long as he did - he continuously refined and developed better intuition for when to buzz.
Many contestants (a large portion of whom likely participated in college Trivia Bowls and were seasoned trivia buffs) could answer the vast majority of Jeopardy clues with relative ease.
The reason for this is that Jeopardy, in order to have household appeal, cannot legitimately engage in an increasingly difficult arms race for its questions - it has to have a capped ceiling in terms of clue difficulty.
That is covered in the article:
> Recently, however, that long-held truism has been shaken: The show has started releasing statistics about buzzer attempts, revealing that some contestants, especially those in the game’s upper echelons, ring in much more often than others. It’s not that the buzzer doesn’t matter—it just matters much less than was previously believed, a shake-up that has reelevated the importance of subject-matter expertise.
> in order to have household appeal
Difficult questions are fun, there's a reason the UK show "Only Connect" does well. Yeah, I'm not going to know what's going on when the question relies on knowing the difference between Finnegans Wake and Finnegan's Wake, but when you list decreasingly obscure eliminated Google products I know what's going on by clue three even if many contestants only even recognize the fourth clue.
Among HN readers I'm guessing "Control" and "Shift" already set people on the correct path before they see "Alt" or "Option" but a lot of word focused people doing such a quiz will be studying the words - counting vowels, thinking about meanings in other languages before they realise it's about keyboards.
Jeopardy already puts a small spin on things with the reverse format, they could work on that harder to deliver a more challenging quiz that's still getting the audience.
I really like NAQT's (National Academic Quiz Tournaments ) take on questions, where it goes from more obscure to less obscure the further you get into the text. For example:
Though he suffered from Bright’s disease, he allowed himself to be considered for his party’s nomination, but he lost to James G. (*) Blaine. Once known as a Stalwart, he declared his independence by signing the popular Pendleton Act. For ten points—name this New York politician who acceded to the presidency upon the 1881 assassination of James Garfield.
The asterisk indicates bonus points if a player correctly answers the question before the host finishes reading past that mark.
But this kind of format where you can "buzz in early" would be a pretty drastic departure for the show.
There were two astounding takeaways when Watson beat Ken and Brad:
1. We've advanced the state of the art in terms of being able to reason and parse complex language to a level the general population wasn't aware of.
2. Beating humans on the buzzer, of all things, feels intrinsically unfair and there's no graceful way to account for that.
Watson's ability to interpret, correctly, what most of the Jeopardy! "answers" were looking for was a sea change. It really was as big as Deep Blue, the crossing of a frontier we collectively thought would likely be much further in the future, and it was here now: complex clues, difficult to reason out context, and the ability to search its corpus of knowledge and assign a confidence value and then produce accurate results... it was mind blowing.
It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now, owing to the specific nature of generative transformers and the advent of attention, specifically. But Watson was pre "attention is all you need!" It was blood, sweat, and tears, and markov chains, and all kinds of craziness. It was a huge swing, and the fact that it actually worked was remarkable.
But back to the buzzer: most people sitting at home probably don't realize that the buzzer is locked out from buzzing in until the host (then Alex, now Ken) completes the read of the clue. At that point, an off-stage producer presses a switch to unlock the buzzers. So prospective players spend their weeks before their tape date spamming a retractable pen in front of the TV trying to nail the timing from the cadence of the host's voice, and hoping they can get it right. Because getting it wrong penalizes you: if you attempt to buzz in before the lock is lifted, your buzzer is individually locked for another 250ms or so. Another producer has a screen that monitors the buzzer activity so they can see if players are habitually buzzing in too early and getting locked, or to monitor for malfunctions, and it's wild to watch it light up like a Christmas tree in the 100 or so last milliseconds of the clue.
This is why you see players spam the buzzer: if they're locked out, the best thing they can do is keep attempting to get back in until the lock lifts. This is also why on rare occasions, you'll see all three players spamming and no one actually getting buzzed in: everyone locked.
So how do you translate this to a computer? The answer was basically to create a makeshift "finger" that would activate the buzzer, but would consistently buzz in at exactly 28ms after the lock cleared... every single time (so long as its confidence was high enough).
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer. It's a signal loop: wait for lock to clear, receive signal lock has cleared, activate buzzer 28ms later. Every device with a stopwatch or timer or clock is more accurate than a human attempting the same timing exercise. It's a solved problem, and not an interesting one. And it disproportionately took the wind out of the sails from the human competitors, who are both the most world-class, best-on-the-planet Jeopardy! players, who were losing by milliseconds to that signal loop.
It's still remarkable Watson was able to successfully answer so many clues correctly and end up winning, but the buzzer aspect was an (intractable) bummer.
If I recall, Watson was fed the whole text of the answer as soon as the other contestants could see it. Personally, I thought it would be fairer to have it do speech-to-text and/or OCR to level the playing field.
But I suppose these constraints are just targeting the machine's input mechanism and not its actual reasoning ability once the answer is read. I'm curious how Watson at the time could handle a particular category that Ken dominated: "Initials to Roman Numerals to Numbers" [1]
[1] https://youtu.be/PsQ_mT5YSvg?si=8L9uKJj1hpYU_kuf
I think it's fair to say that there's nothing, at _all_, remarkable about a computer beating a human on a buzzer.
As anyone who has experienced early video game consoles can attest --- even the <2MHz CPU of a NES is far more than fast enough to beat a human at reaction time, which is what makes it all the more impressive when humans do things like https://news.ycombinator.com/item?id=38841080
>It's funny because, in a pinch, ChatGPT will answer essentially every Jeopardy! clue correctly every single time now
“Answering Jeopardy questions” is one of those tasks where being close to correct is not good enough. A system that hallucinates plausible-sounding answers would be a kind of awful approach to creating software that’s meant to 100% a game like Jeopardy
But a system with a perfect memory and sufficient search space would be great at it. This is precisely where the stochastic database, even a huge Markov model would work great.
Only the hard wordplays maybe would pose a problem, and statistically you can only go so hard on these before human players start lagging too.
Remembering a ton of facts is something the computer would be excellent at...
I mean, it's not "close to correct". It's... really almost 100% correct.
Go try it. Hit up J-Archive, prompt it that you'll feed it Jeopardy answers and want a simple response in the form of a question. I've yet to stump it, even with stuff like Rhyme Time: "a racket-&-net sport peril" (tennis menace) or other more complex clues. Every once in awhile, it'll switch to the thinking model, but most were immediate and correct.
Try here: https://j-archive.com/showgame.php?game_id=8967
Tbf it almost certainly included this whole site in the training data.
ChatGPT would have to just always buzz since generating text is too slow for it to come up with an answer before buzzing.
Watson was given the whole prompt (as text) right when Trebek started reading it out aloud, so it had a full 6-7 seconds to choose an answer.
ChatGPT is more than fast enough to get an answer under the same constraints.
I thought Jeopardy!’s most controversial moment was when Ken Jennings used the word “ho”.
I still think its when they went ultra pedantic about the pronunciation of Solzhenitsyn.
[flagged]
I'm really curious how this kind of writing will be perceived in the coming years. It's "good" long form writing but a lot of that also feels like journalistic padding for the sake of the form, as if you asked an LLM to "write in the style of a long form journalist."
Easy example:
"...dismissed as “disappointing,” “unfair,” and—my favorite—“a gimmick”..."
Is it really important for me to know Clare McNear's favourite? A couple of years ago I wouldn't have thought anything of it but now I'm so sickened constantly by LLMs adding so much needless cruft to everything they write I'm reading this thinking "just tell me the story, for the love of humanity!"
I’m seeing hyperreality in real time. You complain that an article of long form journalism appears to be the sort of thing that might be produced by asking an LLM to create it?
Doesn’t that mean it is true to form? Can you not imagine that some people like reading stories written by others?
> Doesn’t that mean it is true to form?
Yes, that's exactly what it means. Like I say, a few years ago I wouldn't have thought anything of it, but now it's behind painfully clear how formulaic that kind of writing is.
> Can you not imagine that some people like reading stories written by others?
This is quite uncharitable and a straw man. I love reading stories written by others. I'm complaining very specifically about the over reliance on tropes of the form.
I also wasn't leading with the complaint, I was leading with the curiosity I feel about how writing will develop as a result of what I'm seeing here, as I'm sure I'm not the only one who suddenly feels different about human made writing. That's not too say I don't lament it, either, but it is what it is.
I think it's valid to say that the advent of LLMs has diminished the value of formulaic long-winded prose, and increased the value of being brief and concise.
Human long-form writing will be perceived as the top of value.
What is more valued today, IKEA furniture or hand-made natural wood furniture? A mass-produced Timex quartz watch or a hand-made mechanical watch? A painting or an ink-jet print of a painting?
Good, creative, original human long-form writing, yes, I'm sure.
But that's quite different from something essentially indistinguishable from AI output. The article here is very by-the-numbers and doesn't really show any strengths of the human mind over the ability of LLMs, does it? Or can you tell me why you enjoyed hearing which was her favourite quote, and why that makes the article akin to a handmade mechanical watch?