David Ross Sees Room for Improvement in 3D Graphics, Depth Mapping

As interest in live 3D production continues to skyrocket, the topic of depth mapping and its relation to graphics placement has begun to garner particular interest. Graphics placement proved to be a persistent issue for several of the high-profile 3D sports productions thus far in 2010, and the industry has yet to agree on the “appropriate” placement of 3D graphics in within the depth of field.

“Actually producing a depth map is quite difficult today,” Grass Valley CTO Ray Black said at SVG’s first-annual 3D Sports Transmission and Production Summit on May 20. “Although it would work well in postproduction, doing it in real time using multiple cameras in a live switcher for a live event is quite challenging.”

Ross Video CEO David Ross considers the actual 3D graphics thus far to be “phenomenal.” However, he has expressed concerns over the way in which these graphics have been integrated into live 3D productions and sees much work to be done on this front.

“Ross is unique because we’re the only company that I’m aware of that actually owns technology for production switchers as well as technology for graphics,” he said. “One of the big things we’re working on is depth mapping: being able to look at the background not just as a flat video channel to be overlaid on top of. It’s all about the interaction of the graphics and the background video.”

At last week’s 3D Sports Summit, Ross laid out what he sees as the five primary options when it comes to the placement of 3D graphics for live sports productions. Here’s an edited version:

1. Drop the Graphic
You could drop the key entirely; that’s the easiest thing. It’s not the best solution, but it may be better than giving someone a headache or nausea, especially during a long show.

For example, if you’re planning on doing a shot where the action is going to come right out of the screen and it’s going to come beyond the plane of the CG, you drop [the graphic]. You can do that manually if you’re planning it, but you can’t plan it if you don’t really know how the action is going to play out, as in a [live sports production].

However, if you do have real-time ability to figure out the depth map, you can determine when the [graphic] is within the danger zone of the [action]. You can then tell the switcher to drop the key or the switcher tells itself to drop the key, depending on what’s actually doing the depth mapping. So that at least avoids the problem, but then you have the question of when do you bring the [graphic] back? Is that an automatic thing? is that a manual thing? Is that settable?

2. Move the Graphic
If you move the key to follow the maximum depth, it can work, and it’s the most readable thing if you want to see the graphic. As the linebacker comes through the screen, it would be nice to be able to take the graphic and have it jump out of the way. Just move forward and then track back to where it’s supposed to go. Clearly, your eyes are already converging on this guy coming at you, so the graphics could follow and then go back to their spot. You can’t do that if the CG just thinks that it’s on background video. It has to be aware of [the depth map].

This works for something like the scoreboard because people don’t have a problem looking at the score and then looking back at the action. But the problem comes when you’re doing subtitles: you need to be able to watch the action and the [subtitles] at the same time.

3. Intersect or Cut Through the Graphic
This option has advantages and disadvantages. If the linebacker goes right through the scoreboard like a ribbon that he’s cutting through, then, all of a sudden, it’s very cool. It’s comfortable, you’re able to read it, and then boom, the sports figure just goes right through it. It becomes part of the action.

That can also be disconcerting, though. What if you can’t see the score at the moment when you absolutely need to? Or what if that [graphic is] an ad? You’re going to be paying for the fact that the Pepsi ad disappeared because the guy ran through that Pepsi [graphic].

This is also the hardest route technologically. For the other things I’ve been talking about, you may calculate the depth map for the screen, but you’re really only dealing with one number, which tells you how close [the graphic] is. Now you’re talking about accurately getting that number for every single pixel on the screen simultaneously, knowing those same numbers in the CG and merging them and then feathering the edges to make it look real. That’s bloody hard.

If you’ve got subtitles, this is not a good idea. It’s very annoying — like someone putting a hand in front of your newspaper while you’re trying to read it. This [technique] is for a shot clock, a scoreboard, something you know will come back. You’re not reading it; it’s just there for you to refer to.

4. Compress the Background or Push It Back
If the [action] is coming at you and you don’t want to drop the key, you don’t want to move the key, and you don’t want to make people sick, then you can flatten the background in real time. I don’t think the cameras can keep up, but video processing may well be able to. This is the CG talking back to the switcher — interacting in some way — and being able to say, I’m here, you can’t come any closer. So, as the action comes closer, you drop your depth of field in real time.

Pushing the background back is actually a really great idea. When you’re about to bring up the key, you can make sure that the action doesn’t come onto the [same] plane as the key. You’re just trying to tweak it. All you’re trying to do is move it back another 6 inches or so, clean up your graphic, and then drop your graphic and slowly get the full action range back to the background.

5. Just Ignore it
The last option is just to ignore [the problem]. That’s what most of us are doing today in live television. If all you’re going to do is ignore it, then you’ll probably give people a headache, but there’s no technology involved whatsoever, and you just try to position the key the best you can. But, if you ignore it, then it just keeps hitting you, and it can make for [an uncomfortable 3D experience].

We obviously need to have better integration between depth information of what the switcher is switching and what the CG is creating. I’m glad to have both these technologies at Ross Video.