RSS Feed

HCW Tech Blog

For the latest info on computer hardware, tech, news, video games, software tips, and Linux, check out our new improved front page: HCW Tech Blog

By: Bryan Pizzuti [06.06.03]

Discuss this article in the forum!
Registration NOT Required!


Let's face it: Futuremark is a BUSINESS.  Their job isn't to give us a fair benchmark out of the goodness of their heart.  They're out there to make as much money as possible.  That money comes from large companies paying to join their beta program, like Dell, ATI, and, originally, NVIDIA. Not only that but there are different LEVELS of beta program member, dependent on how much money you pay.  ATI is a "strategic" member, whereas S3 and SiS are merely "Active" beta members.  And poor Trident Microsystems and ALi are just beta members.  I'm not these companies; I don't know any specifics on how this beta program works, including whether or not they're allowed to offer code suggestions for inclusion or not. But I do question whether or not it's fair.  Whether or not that company had a better card or better drivers is irrelavent: Futuremark has a vested financial interest in promoting their partners with the deep pockets, especially if they want to see more money later. It might not be fair, but it IS business.

I'm not a business, I'm a journalist. I want fair. I want unbiased. My guess is that most of you readers want that too.  You want to know how powerful a video card is in a quasi real-world situation, where the developers put in optimizations for the major codepaths and GPU instruction sets in use. How do you do that with closed code written by a company unequally making money off of video card manufacturers?  You can't be sure. And ultimately, if they're not making the SAME amount from ALL graphics manufacturers, unfairness will likely leak in. Even the fact that companies with more resources can work more closely with Futuremark to "tweak" things introduces a slight tilt. Tilt isn't too good either.  So what's the resort?  Is there another option?

I know this sounds like a tirade against Futuremark, but it really isn't.  As I said, they are a business, and they're entitled to making money as fits their business model.  What this is is a lead in to a suggestion.

I think what the graphics card industry (as well as us techie-journalists) need is an open source, DirectX9-based benchmark that implements optimized codepaths from EVERY GPU manufacturer. That means the two high performance players (NVIDIA and ATI) as well as all of the mainstream/wannabe guys (Add in SiS). Also, when a new player comes to market (S3: DeltaChrome), they should be given the same advantage as the others, thereby creating a level playing field. If the code is open, no one can sneak anything in (except Microsoft, by sneaking it into DirectX itself, but there's nothing we can do about that).  It's there for the world to see and analyze if it's fair or not, and there's no cases of "well, X paid to be in the beta program; that's why they score higher than Y."  Open source code would allow few secrets: not only would the results be available for public view and interpretation, but so would the code used to GENERATE these results.  The first time someone tries to sneak something into it, someone else will see it, and blow the whistle.

I can't do that; I'm an absolutely ROTTEN programmer with no graphics programming experience.  I wish I COULD do it; I'd certainly open the source up. It's not like it needs to be pretty looking; it just needs to be a fair and comprehensive test.

Right now no games are in the planning process that make full use of all the DirectX9+ features. We have no way of testing performance under FP16, FP24, or FP32 color modes.  We can't test version 2 vertex shaders, nor can we test and compare ATI and NVIDIA's different methods for offering extended length instructions for shaders and their impact on performance. A non platform-specific benchmark for stencil shadows will exist soon, in EgoSoft's X2 rolling benchmark, but other than FableMark, made by PowerVR for PowerVR chipsets, nowhere else that I'm aware of. There's no way to compare classic multitexturing techniques versus using shaders to achieve the same effects.  Things like TRUFORM and N-Patches (curved surface technology) have no real effective tests out there; we had to makeshift something using Serious Sam SE, since that technology (which was introduced back on the RADEON 8500) STILL isn't in use, and we aren't even 100% sure if it works or not if implemented properly, other than a demo released by ATI.

HardOCP had some good ideas as well, which they assembled into a PDF document.  Identical screenshots for IQ tests, ability to create color bars to easily see filtering level boundaries, no auto-tweaking of the program for whatever card is detected (though it might be more beneficial to allow this, and either enable or disable it). I love the built-in image-quality test idea.  If some company is tweaking their drivers to lower quality levels in order to do better in the benchmark program, it would show up in that quality test screenshot.

Eventually we WILL have games that implement at least some of this stuff, and some of the DirectX9 graphics feature set. From there we'll have our game-based tests and benchmarks, which aren't always fair, but they're indicicative of the real world, and they give us our real-world performance numbers.  These tell us you can REALLY expect to see out of that game sitting on the shelf.  But we need something to show us the POTENTIAL of these GPUs too.  Some of this stuff that hasn't been implemented needs to be tested.  We as journalists want to know how well it'll work when developed for properly, so we can give detailed, informed information about it to our readers. We as consumers want to know how well it'll work, since we want to know how long this card will last in our PCs before we shell out money.  Developers want to know so they know whether or not to bother making use of these snazzy new features, and a source other than the GPU provider can always be helpful in decision-making.

The thing is, HardOCP is focused on more real-world benchmarking utilities included with games, which use the actual game engines.  And we definitely need those; don't get me wrong.  But there's also a place for synthetic benchmarks that can let us get into the nitty-gritty of how this card's pixel shader performance is compared to that one, or how effective this one's memory controller is compared to that one. Game-based benchmarks aren't always the best platform for nitty-gritty stuff. Not only that, they're usually restricted to FPS games, usually based on the UT2003 or Quake engines, so we don't get impressions of performance in such genres as racing, real-time strategy, flight sim, and sports games, which are also big sellers. A game that excels in pushing first-person shooter polygons might suck at running a polygon-based real-time strategy game.

Games are a good way to judge performance, since they don't make the bulk of their money from graphics manufacturers.  They make their money from consumers, who own a wide range of cards; it's in their interests to have as balanced a compatibility as the can possibly implement.  But, as I said, there is a place for synthetic tests as well, if one can be found that we can be sure is fair.  And I don't mean one that generates a total score.  I mean something that will let us look at performance of this piece versus that piece, and did this section of the GPU get better with this new generation.  A total score is just a nice little add-on, like icing on a cake (incidentally, I'm not that fond of icing). But what we REALLY need is that little picky data that can show us the difference between one card and another in a way that goes beyond mere FPS scores or 3DMarks. This is the kind of data that will allow us journalists to tell where a card is strong, where it's weak, how long it'll last technologically, how likely it is a function will be used, how difficult it is to code (from the comments of those who know how to read the source), and whether or not you actually WANT to use a function.

However, a fair test is hard to find.  As we said before, to guarantee fairness, it pretty much means an open-source or GPL program, a company that takes equally from ALL graphics manufacturers, or a company that takes NO money from ANY graphics manufacturers.  Companies exist to make money, so the last two aren't terribly likely.  Maybe someone will write something open-source that will let us test all this stuff.  We can only hope that we can have a controversy-free synthetic benchmark, which tests all of those funky features out there, soon.  I join HardOCP in asking for more benchmarking tools in off-the-shelf games, especially for non-first-person-shooter genres.  But I would also like to ask all of the programmers out there who know anything about graphics programming to consider how they want the future of synthetic video benchmarks to look.  And whether or not they want that future to look better.  Just like any other journalist, all I want is the tools to do my job properly, and do a good job at it.  All I ask for is a fair, controversy-free synthetic benchmark.  Is that so much?

Please be sure to visit our forums to discuss any thoughts about this editorial. Registration is NOT required.