Benchmarks

From PEL Wiki

These are a few of the Benchmarks I'd like to have/use

According to JC Mogul's Brittle Metrics in OS research Benchmarks should:

1. Be widely used
2. Be reproducible
3. Predict (in order of descending usefulness):
   1. Absolute performance
   2. Relative performance
   3. Rank order

Contents 1 Memory System Benchmarks 2 Full System Benchmarks 3 Database 4 Model Checking 5 Video Editing 6 File Systems 7 Web Server 8 Super Computer Center Benchmarks 8.1 Center for Remote Sensing 8.2 David Nielson (Physics) 8.3 Jaron Hansen (Chemistry)

Memory System Benchmarks

SPEC CPU might be interesting here because it exercises the RAM. Since they never go to disk, they'd be most effective at helping me make sure that I wasn't killing performance in a silly way outside of the slowdown I'll get from going to disk frequently.

Full System Benchmarks

Futuremark has some benchmarks like PCMark and 3DMark which are supposed to exercise the whole or parts of the system.

Database

There are lots of benchmarks which incorporate databases. I'd like to find one which is easy to configure, uses a large database, and does something (or pretends well that it is doing something) useful. In order to get the full benefit of DiskRAM, I think the database would have to be rewritten to take the persistence of the storage into account. Maybe I'll use a RAM database which doesn't worry about persistence.

Micro Memory is a company that builds battery-backed NVRAM products for accelerating databases.

Model Checking

Tonga's Disk-based Model Checker

The idea is that model checkers need large state spaces.

For some reason, it doesn't run on my 64-bit system unless it's compiled on my 32-bit system. I guess I need to track that down.

Video Editing

I think I can find a widely used video editing system, but I haven't found a benchmark which is widely used in the literature. (Unless you count PC vs. MAC tests as "the literature")

One of the problems seems to be that the benchmarks are relatively small (e.g., how long does it take to encode this 50 MB file)

File Systems

File system benchmarks seem to be either microbenchmarks or traces of actual usage. I'd like find something which is an application which exercises the filesystem well. Maybe I should set up the machine to be a file server on a local network?

Web Server

Here's a quote from an IBM Tech Report

For example, the Apache webserver is notoriously ineffective on SPECweb99 since it uses the operating system's file caching mechanism instead of supplying its own. This means that each time an infrequently accessed and very large Class 3 file is brought into the cache, many small, frequently accessed files are discarded and thrashing results.

DiskRAM should eliminate the need for everyone to make their own cache. The problem is designing a good study to show that.

Super Computer Center Benchmarks

These are a few users that prefer larger memories to more processors around campus. It would be interesting to team up with some of them.

Center for Remote Sensing

MERS from EE. I haven't talked to them yet, but they have a lot of geospatial data.

David Nielson (Physics)

Jaron Hansen (Chemistry)