Illustration by Mike McQuade
Truth No. 1: Most of us watch sports to see the unexpected. Truth No. 2: Plenty of us want to predict the future.
Somewhere, where those two contradictory truths meet, there has been a movement afoot. For decades now, sports-crazed statheads — the sabermetricians and forecasters and moneyballers bent on winning their fantasy leagues, assembling an actual professional team or simply understanding the sports world — have been honing their techniques, trying to find the signal hiding in the noise. In baseball alone, an alphabet soup of player projection systems have been born — ZiPS, CAIRO, CHONE. We just introduced CARMELO to basketball. The movement is trying, in other words, to predict the unexpected.
There are some in the movement who want to project the future, quite literally, on the screens in front of our eyes. Somewhere in the foothills of Oregon’s Cascade Mountains, an astrophysicist and his son are working, with the backing of an outspoken billionaire, to bring us just such a glimpse. Armed only with a camera, a laptop and their custom code, they’re working on a system that calls a 3-pointer a swish or a brick, a volleyball serve in or out, a soccer shot over the bar or in the goal, all before the ball completes its flight. If the system works — and that’s a big “if” — it would be equivalent to a minor superpower: flash precognition. The sports fan would become, if only for a second or two, a superhero.
And the system is almost done. This, right here, could be the future of sports:
The footage is from a Dallas Mavericks game against the San Antonio Spurs in March. What you see was calculated in real time, but for demonstration purposes the shot itself is slowed down. A computer tracked the ball’s position as well as its projected position, and the three red bars underneath the action indicate the system’s confidence that the ball would miss left, go in, or miss right, respectively. In this clip, it was a Monta Ellis jumper that went in, just as the tech predicted.
“Many decisions in sports relate to the trajectory of a ball or similar object, such as a puck or shuttlecock,” reads the patent application for this technology filed in late August. There are three names on the patent application: Matthew L. Ginsberg, Navarre S. Ginsberg and Mavericks owner Mark Cuban. The Ginsbergs have ambitions to spread their technology far and high, including to the NBA and its billionaire owners, including Cuban.
When Matt Ginsberg first mentioned this project to me, back in March, he began the conversation like this: “I’m going to revolutionize sports.”
His idea is simple: Find a ball with a camera and have it tell a computer what’s up with the ball (or shuttlecock or javelin or frisbee or whatever). Then have the computer calculate, in real time, where the ball’s going. Then turn that into some useful piece of information, knowing what sport we’re watching and the dimensions of that sport’s infrastructure — lines on the ground, baskets in the air, and so on. Have the computer tell you, maybe along with some measurement of its certainty, “that basketball will go in the basket” or “that volleyball will land outside the lines.”
Then do something interesting with that fact. Have a red light go off to signal an out-of-bounds serve to the returning team. Have a soccer goalie’s smartwatch buzz if a shot is going to clear the bar, telling her she needn’t parry it and concede a corner kick. Put it on the TV screen for the folks at home.
Again, the idea is simple. Almost comically so, judging by illustrations in the patent application.
The execution, on the other hand, is not simple. Matt Ginsberg’s training is in astrophysics. He got his Ph.D. from Oxford when he was 24 years old. His doctoral advisor there was the famed mathematical physicist Roger Penrose, and he recalls rubbing elbows with the academic rock stars Stephen Hawking and the late Richard Feynman. He created an artificial intelligence crossword puzzle solver called Dr. Fill and a computer bridge world champion called GIB.
Unsurprisingly, there’s pretty heavy math involved to make this real-time sports predictor work. For one element of the system’s calculations, Ginsberg sent me a pdf with eight dense pages of physics diagrams and systems of equations and notes on derivations. It uses something called the Levenberg-Marquardt algorithm. It requires Jacobians and the taking of partial derivatives and the solving of quartics, and code efficient enough to calculate it all up to the split second. If predicting the future were easy, I suppose everybody would do it. (See Truth No. 2, above.)
One thing this project can’t predict, however, is its own future. Its uses are, so far, largely speculative, and cashing in on a minor superpower might not be easy. Even gamblers who bet during play would struggle to make much money from a half-second heads-up that a shot is going in. But Ginsberg’s system would find a natural place in the long line of sports technologies that have been used for a singular end — TV.
The vast majority of our collective sports-viewing is on television. Around 21 million people watch an average Sunday Night Football game on TV, for example — some 300 times more than the 70,000 who are able to see it in person. Our sports experience is, to a first approximation, a television experience. I’ve seen Tom Brady play dozens of times, even though I’ve never seen Tom Brady play.
And television has been enhancing — or, at the very least, altering — how we watch sports ever since TV was invented. NBC coverage of the 1936 Olympics in Berlin, the first sporting event ever televised, used slow motion footage to “show the form, the poetry of motion” of a pole vaulter.
It seems a natural fit. Cameras and technology can do many things our eyes can’t. If we can see closeups of Pluto, surely we deserve a crystal-clear view of Odell Beckham Jr.’s catch. “Keep your eye on the ball” is the child’s earliest and most universal sports lesson. And nowadays we can see just about every little thing that happens to the ball. Or puck.
Nearly 20 years ago, on Jan. 20, 1996, at the NHL All-Star Game, FoxTrax made its debut. FoxTrax is better known as a glowing hockey puck whizzing around the screen.
The development of this tech, which looks quaint in retrospect, was a major undertaking. In 1994, an executive vice president at News Corp. promised to develop glow puck technology within two years, for $2 million, according to a 2003 article in IEEE Spectrum. He scooped up a team of 10 with military engineering experience — in radar, underwater sensors and radio-positioning systems — and sought outside help from other defense engineers. It was all hands on deck to track a hockey puck.
But the system was discontinued after three years. FoxTrax’s main problem was probably aesthetics. It was distracting, and the puck’s “tail” looked better suited to a comic book than a hockey game. Hockey fans protested, the broadcast rights changed networks, and the phenomenon died.
But its developers were undeterred. They turned their attention to a problem that sounds easier, but was much trickier. A couple years later — on Sept. 27, 1998 — the middling Cincinnati Bengals and Baltimore Ravens met in Baltimore. At 8:20 p.m. local time, a technology called 1st & Ten debuted. It’s better known as the yellow first-down line. The yellow line isn’t official — as anyone who’s ever watched a football game on TV could tell you — but the yellow line is beloved. I had 13 good football-watching years under my belt before its introduction, but I can’t remember watching a single game without it. The yellow line is ubiquitous. The yellow line won an Emmy. The yellow line is here to stay.
Technologies like these told us more about what we were looking at by putting a visual layer between us and a game on our TV. FoxTrax told us where the puck was at all times. 1st & Ten tells us where a team is trying to go. But they were just building blocks. Data was the next frontier.
A torrent of new innovations followed in their wake. The NFL and Zebra Technologies have strapped radio-frequency identification chips onto players this season. The camera-tracking system SportVU has been hailed as the future of the NBA by our friends at Grantland. ProTracer technology has given golf fans something to stare at other than the warm plasma-screen glow of the summer sky. Hawk-Eye technology in tennis powers replay challenges and can track a ball to within mere millimeters. LiveLine, another Emmy winner, does its best to make sailing interesting to watch. And one word — in press releases, company websites and media coverage of these technologies — appears over and over again: “revolutionary.” Layering data on top of a sports broadcast is the frontier.
But, as with most revolutions, there is a staunch establishment that leans against the shifting winds. In April, Vice published a philippic against K-Zone, the imaginary strike zone projected on the screen during baseball games. “The calculus at the root of this experiment seems to be that we prefer perfect information to beauty, precision to custom,” Robert O’Connell wrote. And some even rebel against television itself. Each season, the supremacy of radio-baseball to TV-baseball is vocally declaimed by acolytes. “Listening to a game on the radio, while driving along through the night hits some sort of cosmic level of perfection, especially if you can find it on an AM station, with a slight whine from some other signal, scratchy static calling the game in from across time and space,” Todd VanDerWerff, Vox’s culture editor, wrote in his newsletter earlier this month. “The fall of baseball could certainly be tied to the slow decline of radio as well,” he added.
The natural-human-beauty-vs.-cold-mechanical-statistics sports debate has been thoroughly litigated, including on FiveThirtyEight. The jury is hopelessly hung. Do you want a dressed-up broadcast? Do you want a television screen augmented with pitch counts and wind speeds and strike zones and Bryce Harper’s velocity running to first? Or do you simply want to tune your dial to AM 720 for the crack of the bat and the passionate, pained voice of Ron Santo, may he rest in peace?
I’d guess the split is largely generational. As the aesthetics of real televised sports approach those of sports video games, with their elaborate heads-up displays and options, the younger set may feel more at ease. But there’s more than just aesthetics that sports share with video games. The outcomes of events in both are pre-known, if you know where to look. When you kick a field goal in Madden 16, for example, the path of the ball is already written. Sure, you’ll see the ball fly through the air for a few seconds, and perhaps drift slowly toward the right upright, causing you to clench. But the game and your Xbox already “know” if it’s good or wide right — the kick’s power and distance, the wind, etc. have already been thrown into whatever algorithm and the result already spat out. The anticipation is just an illusion. But isn’t that the same in real life? When Butler’s Gordon Hayward launched the shot that would’ve beaten Duke in the 2010 NCAA final, it hangs in the air for-seemingly-ever — in fact it’s just shy of two seconds — and we don’t know whether it will go in or out. (See Truth No. 1, above.) But the universe “knows.” Physics “knows.”
Matt Ginsberg’s technology may be able to tell us mortals what the universe already knows — it may let the universe whisper in our ear. We may not have to wait for a resolution. We may, for example, have been able to hear Cinderella’s death knell just a little bit sooner. Rather than “keep your eye on the ball,” it’s now “keep your eye on where the ball will be.”
Sportvision — the company behind football’s 1st & Ten, baseball’s PITCHf/x, sailing’s LiveLine and other tech — has undertaken some real-time projections of a different sort. It has tech that tells TV viewers when one car is expected to pass another in NASCAR, for example. But Hank Adams, Sportvision’s CEO, told me he wasn’t aware of any other technology like Ginsberg’s. It seemed reasonable. “I wouldn’t be at all surprised if he could predict, with some accuracy, whether a ball was going to go in or not,” Adams said.
Adams also mused on the implications of Ginsberg’s technology. Its use as a TV storytelling tool may be limited, he said, given the mere second or two that it allows us to see into the future. He was also skeptical that the NBA would allow any in-game use by teams. It could be a valuable coaching tool, he thought. Or in training. Perhaps in a golf telecast. Maybe for players in a volleyball game. Neither of us was really sure.
Matt Ginsberg is tall and fit with sharp features and, aside from his closely cut grey hair, resembles a 40-year-old rock climber more closely than the 60-year-old technologist and businessman that he is. He’s affable but deeply serious. I first met him in Stamford, Connecticut, in March, at the American Crossword Puzzle Tournament, where he was operating Dr. Fill, his artificial intelligence crossword puzzle solver. In the crossword community, he’s both loved and hated — he’s the guy who might be a little too clever for his own good who’s trying to ruin all the fun with his fancy computer program.
His would-be revolution in sports technology grew out of his role as unofficial statistical analyst for the University of Oregon’s women’s volleyball team. He has, among other things, imported basketball’s adjusted plus-minus system to volleyball, and convinced the team’s coach that the way timeouts were traditionally used was inefficient. Last November, while Ginsberg was watching a game, a player hit a serve that, from Ginsberg’s bleacher seat, looked like it was sure to go out. The returning players should’ve simply let the ball go out but they didn’t. Ginsberg was annoyed. “I can fix this. We can have a computer help,” he told me. “I did not realize how hard it would be.”
While the development of FoxTrax and 1st & Ten resembled military contracts, Matt Ginsberg’s purported crystal ball was developed in a son-and-pop shop in Eugene, Oregon. Navarre Ginsberg is a 21-year-old programmer and Matt Ginsberg’s son. When I reached Navarre Ginsberg by phone in early October, his dad told me not to take up too much of his time — he had to get the camera working. It was the younger Ginsberg who first suggested to his dad that this technology could be taken far beyond just volleyball. Matt is in charge of the big picture; Navarre is responsible for handling coding issues as they arise, and making sure the damn thing works.
The result looks like this. Here’s a Rajon Rondo shot that misses right — as correctly called by the computer:
But the system’s not perfect — not yet. It occasionally doesn’t even recognize a shot is happening, or it thinks a pass is a shot, or it simply makes the wrong call after identifying a shot. Here, it thinks a long pass is a long shot:
The Ginsbergs are aware of their system’s imperfections, but they share an enthusiasm for what it can become. And they want to get it out into the world, perhaps as soon as this NBA season.
“If we haven’t figured out why that’s valuable to a sport yet, we just haven’t thought hard enough yet,” Navarre Ginsberg said.
Looking for investors, and an eventual outlet for his project, Matt Ginsberg approached Mark Cuban, the billionaire owner of the Mavericks, in January. The idea had applications beyond volleyball, Ginsberg had realized. Cuban was hesitant, Ginsberg said, until Ginsberg proposed a deal: Give me $50,000, he said, and I’ll develop it, and the Mavericks can use it in one game.
“$50,000 to win an NBA game of your choice is incredibly cheap,” Ginsberg recalled telling Cuban. “And you don’t care about the 50 grand but I do. And I’ll also give you a right of first refusal across the NBA.” Cuban wanted two years, and Ginsberg could keep the right of first refusal. Ginsberg agreed. A lawyer came in to iron out the finer points of the deal. The lawyer was suspicious. What the hell were the Mavs even buying? It could be unicorns.
Cuban described his involvement in the project to me as “active” — providing tech and design recommendations. But Ginsberg is the brains behind the coding, he said in an email last week. The most promising use of the project, in Cuban’s view? “Real-time predictions on court that can be relayed to the sidelines.” He wouldn’t comment on how the Mavericks intend to use it, if at all.
When I asked Cuban how he thought the NBA would respond, he deflected a bit: “It will have amazing real-time applications in the future — things like detecting whether or not a shot was goaltended, in real time, and relaying that information to officials or displaying it on the backboard.”
Ginsberg’s views on the technology’s uses have been evolving dramatically since we first talked in March, but they’ve always been broad. Some uses seem doable; some no doubt pie-in-the-sky. Goaltending, as Cuban suggested, is one humble but useful application. The technology could ensure that goaltending is always called correctly — it analyzes a ball’s arc, so finding the apex of a given ball’s trajectory to check for goaltending would be easy pickings. Another use is volleyball serves. A system like this is legal in NCAA volleyball — or at least it’s not illegal. Yet. (The Ginsbergs are unabashed Oregon Duck homers. “I’m excited about helping my team,” the elder said.) Another is for soccer goalies. The tech could prevent them from ceding unnecessary corner kicks. Another is tennis. Tennis players could train with the technology, and learn in real time what types of passing shots they should let go at the net and which they should go all out to try to volley.
But the killer app, in many of our conversations, has been basketball tactics. Imagine, Ginsberg would describe, if the home team’s players knew when their opponents’ shots were going to go in. They’d be signaled — a flashing light, maybe — and most of them could immediately race down to their offensive end, knowing they needn’t play any more defense on that play. A huge advantage; a sea change in basketball strategy.
Now, whether that’s practical or would be allowed by the NBA seems questionable, at best. And Ginsberg has backed off this idea somewhat. At the very least, he doesn’t want this tactic available to just one team.
“I don’t want to have every basketball fan who doesn’t live in Dallas hating me,” he said. “That would not make my life better.”
So what about TV?
“There are going to be media applications that I can’t predict, because I’m not a media guy,” he said. “The other thing that’s really become apparent to me, as we’ve gotten closer here, is that I don’t know what I’m doing. In the sense that there’s huge economic value to this. If we can make basketball more fun to watch on TV, how much is that worth to NBC? And I am completely clueless” — so clueless he didn’t realize the NBA hasn’t aired on NBC since 2002.
Watch FiveThirtyEight’s Oliver Roeder and Jody Avirgan talk about Ginsberg’s invention.
This technology’s future may become a lot clearer very soon. Ginsberg has been taking meetings over the phone. This month, he talked with an NBA executive vice president to discuss what impact this technology should have on the game. And he talked with Marc Lasry, the billionaire hedge-fund manager and co-owner of the Milwaukee Bucks, who Ginsberg thinks can help advise him on the economics.
But here’s the thing about predicting the future: You’d better be right. In the 13-minute video Ginsberg sent me, the computer was right on 23 of its 30 calls — about 77 percent accuracy. It also didn’t recognize a shot, or thought a pass was a shot, on 10 occasions. Even just miscalling a few shots in a game could doom a project like this. If this tech is ever integral to the game — for a broadcaster or a pro team — it’ll be a fine line between the computer as Oracle of Delphi and the computer as useless hunk of junk.
The Ginsbergs know this, and have been so busy hammering away at the last pesky nails sticking out of their project that they haven’t even named the thing yet. The patent application calls it Real-Time Sports Advisory System Using Ball Trajectory Prediction — and RTSASUBTP doesn’t exactly roll off the tongue. My suggestion: The tRuth. The technological Babe Ruth. He called his shot, after all.