tag:blogger.com,1999:blog-3821464279005499761.post9068428667712568769..comments2023-11-02T10:58:05.355-04:00Comments on IBM Research: Watson’s wagering strategiesChristopher Sciaccahttp://www.blogger.com/profile/00780044156231457785noreply@blogger.comBlogger25125tag:blogger.com,1999:blog-3821464279005499761.post-92201029205665146092011-11-26T07:53:10.504-05:002011-11-26T07:53:10.504-05:00Excellent :-)Excellent :-)Stiffmaesterhttps://www.blogger.com/profile/15100677996605463327noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-54014832105681820262011-03-03T11:47:34.536-05:002011-03-03T11:47:34.536-05:00is Pascal Wager a good strategy to trick God into ...is Pascal Wager a good strategy to trick God into letting you enter heaven?Brad Fallonhttp://www.facebook.com/bradfallonnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-90249589426214847592011-02-20T20:23:41.039-05:002011-02-20T20:23:41.039-05:00Excellent word you've done boys... I think som...Excellent word you've done boys... I think something particular, but in general is OK...<br /><br />When do you think we can realistically expect to see blog posts from Watson himself (at least genuine capability)? Not self aware yet, but comment and summarize his own experiences and what he (it) learned from interacting with humans? I think if you add a "heartbeat" to Watson constantly interacting with people and learning from them with a purpose of better understanding people it may even get one step closer to being truly self aware (of course depends on algo limitations inside).colesterol altohttp://colesterolalto.infonoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-87383462017525462372011-02-19T08:43:32.717-05:002011-02-19T08:43:32.717-05:00@ Jay-Milwaukee
1. There's a bunch of info out...@ Jay-Milwaukee<br />1. There's a bunch of info out there about the buzzer timing, though I haven't looked at it yet. Whatever the answers, this was intended as an exhibition, so I don't see a need to be "fair", but more importantly, "fair" cuts both ways. Human players can read the clue themselves and begin their thought process as soon as it's revealed. There's a limitation on how soon players can ring in, but they can have an answer sooner. Sending the text file when Alex finishes the question might be considered unfair to Watson.<br /><br />2. Why do you think Watson was "so accurate" in picking daily doubles? Very little about Jeopardy is random, but let's imagine it is. In that case each of 3 players could be expected to get 2 of the 6 daily doubles in the two games and choose 40 of the 120 total questions.<br /><br />Now consider the progress of the game. When the 2nd round starts 1 of 3 players has already found the first DD. Let's assume the 2nd DD is found by a different player. There's now a 2 in 3 chance that the final DD will be found by a player who found a previous DD and a 1 in 3 chance it will be found by the remaining player. That means that each player finding their "fair share" of DD's is only half as likely as one player finding at least twice their "fair share". <br /><br />Now consider that the game isn't random. Unless each player is equal one of them should do better than the others. If one player is doing well and choosing more than 1/3 of the questions they're more likely to find each DD than either of their opponents. I forget what the sequence was for the two games, but Watson certainly chose more than 1/3 of the questions. We already know that the most likely outcome is that one player will find at least 2 of the DD's so it shouldn't be too surprising if it was Watson.<br /><br />So what was the probability of Watson's success at finding DD's? Ken Jennings got at least one DD, and I'm almost certain Brad Rutter got at least one, so at most Watson got 4 of the 6. Without an exact count, I'd guess that Watson chose at least 50% of the questions, so we could expect Watson to have gotten at least 3 of the 6. If Watson chose 55% of the questions we could expect him to have found 55% of daily doubles, but it's impossible to actually find 3.3 of them. In that case the result HAD to be statistically improbable, but only a little bit improbable. OTOH, if Watson chose at least 80 of the 120 questions 4 of the 6 DD's is exactly what we should have expected. In the end there was absolutely nothing unusual about the result even if it was slightly improbable.<br /><br />3. In the first game, even with a high confidence in a correct answer, a small wager for FJ guaranteed a substantial lead after the first day. That seems like a sensible strategy for any player with a big lead and so-so confidence in a correct answer.Stevenoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-67620234665607329922011-02-18T17:25:20.386-05:002011-02-18T17:25:20.386-05:00all I think about when I read about this is : T2.all I think about when I read about this is : T2.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-75104751513796582032011-02-18T09:27:39.082-05:002011-02-18T09:27:39.082-05:00@Abeat:
If Ken had elected to double his money, he...@Abeat:<br />If Ken had elected to double his money, he would have had a 2-day total of $41200.<br /><br />If Watson had answered incorrectly, his 2-day total would have been $41201.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-12455845992726002182011-02-17T17:15:38.232-05:002011-02-17T17:15:38.232-05:00Please view the replay from today's TEDtalk wi...Please view the replay from today's TEDtalk with members of the IBM Watson team for more information on Watson - http://www.ted.com/webcast/archive/event/ibmwatson<br /><br />Kevin<br />EditorKevin Winterfieldhttps://www.blogger.com/profile/16239982867762776783noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-31841474884971854842011-02-17T16:35:02.213-05:002011-02-17T16:35:02.213-05:00I am very skeptic about Watson. A few questions:
1...I am very skeptic about Watson. A few questions:<br />1) We know that the question is transmitted via an interface to Watson. So my question when is the transmission made? In a nano second after the question is posted? So by the time humans read and understand the question, a computer already has an answer. Is that fair?<br />2) How does it so accurately pick out the Daily Doubles? This leads me to think that the whole concept of Daily Doubles not being random needs to be re-evaluated.<br />3) Game 1 Final Jeopardy - Watson loses and the bet was pennies while Game 2 Final Jeopardy - Watson has a huge lead, yet bets a substantial amount and wins. So my question: Is Watson placing a bet before the question is revealed like all humans or after? <br />I might sound like a conspiracy theorist but I am not. Just trying to clear some confusion.Jay-Milwaukeenoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-11475007119815061372011-02-17T13:35:27.268-05:002011-02-17T13:35:27.268-05:00I didn't quite understand that final bet eithe...I didn't quite understand that final bet either. Watson had such a large lead, there was no reason to bet anything. Also, he could have bet all of his points because the closest that the 2nd place person could come was like 32K which was still smaller than the first day total for Watson. I think he was going for brownie points in winning the round (that day's game). It may appear to be poor sportsmanship to pile it on like that, but it did provide a chance for the other players to win the round. None of this probably had anything to do with the calculations - just my anthromorphisizing my new found hero - Watson.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-29346936121070750022011-02-17T11:35:25.762-05:002011-02-17T11:35:25.762-05:00@Rick Carter - Not that I know for certain, but I&...@Rick Carter - Not that I know for certain, but I'd wager (but only in even $100 increments - I am human after all) that Watson does, in fact, continue refining it's answer because I did notice a few instances where Ken or Brad would ring in, and while they were answering, Watson's "guesses" would alter once or twice in that time.<br /><br />The thing I was wondering throughout, since Watson was SO quick to be able to ring in, was at what point was it fed the question: as soon as the text on screen was revealed and Alex started reading; some time in the duration of the reading; or not until Alex finished reading. I couldn't help but think that it was being fed the text for the question as soon as it was revealed and Alex started reading, which if true gave Watson an "unfair" (or at least unrealistic) advantage. A computer can consume and begin processing a text file virtually instantaneously. The humans were limited to reading and/or hearing the answer read, and only at some point a second or two later would they have consumed enough information to begin processing.Unknownhttps://www.blogger.com/profile/02558786152750468554noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-63835308866301330842011-02-17T11:35:12.030-05:002011-02-17T11:35:12.030-05:00I'm a little confused about Watson's wager...I'm a little confused about Watson's wager last night...It seems that he wagered enough so that if he got the question wrong and Ken had doubled his score then Watson would've lost. I understand that Watson takes into account his confidence in the category, other players' bets, and his goal of winning the game. Given his bet, he was obviously confident in the category. However, even if he was 99.999% sure of getting it right, couldn't he have wagered 0 dollars and guaranteed himself a win? <br /><br />I'd understand his wager if the goal was dollar maximization but essentially all dollars were relative last night as each player was awarded a dollar amount based on where they finished in relation to other players. Given this, why did he choose such a wager?Abeatnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-91242430258726555102011-02-17T11:27:32.311-05:002011-02-17T11:27:32.311-05:00Very Interesting blog post, thank you.
Wagering w...Very Interesting blog post, thank you.<br /><br />Wagering was the one place where I felt Watson didn't pass the Turing test during the game. (In other words, could Watson pass for a human player to another human judging only on its responses made during the game?) <br /><br />I think even Watson's flubs could appear as human mistakes if one only read a transcript of the games without any text mentioning Watson as a computer. Even the final jeopardy US Cities answer of 'Toronto' appeared to me like a joke response from someone who didn't know the real answer.<br /><br />But Watson's wagers were the one thing that seemed distinctly non-human to me. To me, the wagers gave the best clue that Watson's intelligence was not human.<br /><br />I think that's why the audience would chuckle a bit whenever Watson made wagers. The wager values seemed like something a machine would calculate rather than values human intelligence would calculate. It was like a little slip up, where Watson revealed itself as non-human.<br /><br />But then again, I still have my doubts about Ken Jennings being a real human, too. :-)Unknownhttps://www.blogger.com/profile/04383998700797067144noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-86359777602879249482011-02-17T10:47:24.205-05:002011-02-17T10:47:24.205-05:00I'm really interested in Watson's "bu...I'm really interested in Watson's "buzz threshold" -- are the algorithms similar to the wagering strategies? I took a stab at it on my blog (article is linked); am I close?<br /><br />Cheers,<br />DavidDavid Roherhttp://harvardsportsanalysis.wordpress.cokmnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-44200776299153025032011-02-17T10:16:59.542-05:002011-02-17T10:16:59.542-05:00I'm willing to wager its more likely a random ...I'm willing to wager its more likely a random number generatorAnonymousnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-25769784814057752502011-02-17T00:53:54.195-05:002011-02-17T00:53:54.195-05:00Fantastic comments.
Please keep posting your comme...Fantastic comments.<br />Please keep posting your comments and questions. We will be using some of them at the TED.com LIVE event “Final Jeopardy and the Future of IBM Watson” event tomorrow 2/17 at 11:30 am ET.<br />Please tune in at http://www.ted.com/pages/view/id/593.<br />Thank you,<br />Kevin Winterfield<br />EditorKevin Winterfieldhttps://www.blogger.com/profile/16239982867762776783noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-51108171875333272912011-02-16T22:47:18.914-05:002011-02-16T22:47:18.914-05:00I found Rick's comment about not having enough...I found Rick's comment about not having enough time to complete the processing interesting. In show #3 you could clearly see the human contestants buzzed in before they had completely determined their answer. Perhaps a strategy around probability of getting this question correct to ring in early. [Maybe similar to the betting heuristics.]<br /><br />All in all - AWESOME performance by Watson and the human players.srlnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-38103919865603651232011-02-16T16:12:04.729-05:002011-02-16T16:12:04.729-05:00I'm curious on a similar but broader question ...I'm curious on a similar but broader question than @kamlesh asked... if Watson has more time in general, does he use it? e.g. someone beats Watson to the buzzer and then gets it wrong -- is Watson still refining results? Similar question for the Final Jeopardy question and the final wager.<br /><br />I'm thinking along the lines of some chess-playing computers where the more time they are given the further through the possibilities they search.Anonymoushttps://www.blogger.com/profile/18202830573232043611noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-15425589021051238082011-02-16T12:42:50.366-05:002011-02-16T12:42:50.366-05:00@SteveK:
I can't speak for the Watson team, b...@SteveK:<br /><br />I can't speak for the Watson team, but I highly doubt Watson is designed in such a way that he can produce interesting blog posts. I'm sure the closest thing he has to a memory are log files listing the various heuristics that offered up possible answers and their scores, as well as various other debugging information - fascinating stuff to us programmers, completely incomprehensible to the untrained, and chock-full of trade secrets :)bd_https://www.blogger.com/profile/01560746058527855728noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-34423353991258462522011-02-16T12:01:00.526-05:002011-02-16T12:01:00.526-05:00When do you think we can realistically expect to s...When do you think we can realistically expect to see blog posts from Watson himself (at least genuine capability)? Not self aware yet, but comment and summarize his own experiences and what he (it) learned from interacting with humans? I think if you add a "heartbeat" to Watson constantly interacting with people and learning from them with a purpose of better understanding people it may even get one step closer to being truly self aware (of course depends on algo limitations inside).SteveKnoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-50179690404223164812011-02-16T10:56:10.040-05:002011-02-16T10:56:10.040-05:00Does Watson start gathering related data as soon a...Does Watson start gathering related data as soon as FJ topic is shown or waits through the commercial break and for the answer (clue) to be shown before starting?Unknownhttps://www.blogger.com/profile/02277052363142903488noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-68131939598018165592011-02-16T09:36:49.778-05:002011-02-16T09:36:49.778-05:00This is amazing! Last night we were laughing at th...This is amazing! Last night we were laughing at the $1246 wager, and so I'm happy to know how it came about.Jennifer Sussinhttps://www.blogger.com/profile/18095301077652066733noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-4395087773764830622011-02-16T09:08:10.579-05:002011-02-16T09:08:10.579-05:00I thought I discovered a really nice restaurant a ...I thought I discovered a really nice restaurant a few days ago. Now that I've read the previous comments I see that I was mistaken.<br /><br />Is there any chance IBM can develop a method by which people with internet access can learn to understand human language?Stevenoreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-62639070436374047912011-02-15T03:13:48.333-05:002011-02-15T03:13:48.333-05:00Great article, Gerald! As you can see from the co...Great article, Gerald! As you can see from the comments, this is an area of intense interest among hardcore Jeopardy fans. I found it interesting that many of your strategies confirmed my own thoughts about how to attack aspects of the game. It demonstrates that the same good ideas can be developed independently by different people. I'm sure that's also the case with the "first equals second plus third" scenario. I've spent a lot of time on the J! Archive and I think it's great, but I was not aware of that scenario since it is not specifically documented on the site. The site's wagering calculator may have 129 scenarios, but to the user, it tends to be a black box. It is not always clear what happens between the input and the output.<br /><br />I agree with Robert that it would be interesting to read a lot more detail about the strategies that Watson uses. I think some of Watson's techniques may influence the way humans play the game in the future. Robert is justifiably proud of all his work to document and preserve Jeopardy! history. I hope the Watson team will be equally generous in sharing what they have learned.Ed Toutanthttps://www.blogger.com/profile/11380695845136853988noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-1376606130995945672011-02-14T16:34:55.354-05:002011-02-14T16:34:55.354-05:00It would be interesting to know which novel wageri...It would be interesting to know which novel wagering scenario situations the Watson team discovered. The "first equals second plus third" scenario described in the footnote is not one of them, having been known to Jeopardy! enthusiasts and implemented in the J! Archive wagering calculator for many years, predating the Watson project. The wagering calculator, which does not use any machine learning or otherwise statistics-based analysis but instead consists of a very large if-statement tree, has identified over 129 basic wagering scenarios, and there are many that properly call for a bet to tie. Among them:<br /><br />A = 2(B - C) (the "Faith Love" scenario)<br />Exact fractions (2/3, 3/4, 4/5...)<br />A = 2B - C ("Evenly spaced scores")<br />A = B + C ("First equals second plus third")<br />A = B + C/2, B != C ("First equals second plus half of third")<br />A = 2C, C < (2/3)B ("First is twice third, with third less than two-thirds second", discovered by Jeopardy! Message Board user slam)<br />A = 2C, B = 2C ("first and second both twice third")<br />B + C = 3/2 A ("Second plus third equals three-halfs first", discovered by Jeopardy! Message Board user Gneq with additional conditions suggested by Jeopardy! Message Board user K703)<br />A = B = C ("Three-way tie")<br />A = B, C > A/2 (The "tortiose and the hares" scenario)<br />A = B, C < B/2 (The probably mis-named "Prisoner's dilemma" scenario)<br />All the various "lock-tie" scenarios where A = B/2<br /><br />Hopefully, the Watson team will find proper occasion and forum to publish their wagering findings in full, as they would be beneficial to all Jeopardy! enthusiasts and game theory buffs.<br /><br />All the best,<br />Robert K SRobert K Shttps://www.blogger.com/profile/13320641533825974319noreply@blogger.comtag:blogger.com,1999:blog-3821464279005499761.post-3462407262104302382011-02-14T16:16:09.887-05:002011-02-14T16:16:09.887-05:00Claiming discovery on first = second + third? That...Claiming discovery on first = second + third? That's the only major flaw in the article -- that concept has been well-known in Jeopardy! circles for years. Otherwise, it's great.Anonymoushttps://www.blogger.com/profile/12650546197053700213noreply@blogger.com