~/imallett (Ian Mallett)

The F32 High Dynamic Range File Formats
Invented and Written by Ian Mallett

Table of Contents
Introduction
Motivation
Documentation
Example Data
Future Work
Questions and FAQ

Introduction

F32 is an extremely simple class of high dynamic range image storing file formats, focusing on ease-of-implementation. It is a 100% open standard, in the sense that anyone is free to implement and use the format. An attempt to fully document it is provided here as well. The extension ".f32" clashes with an obscure audio format, but I doubt this will cause any confusion.

Motivation: Why Do We Need Another HDR Format?

Every HDR format that I am aware of is complicated by some slick feature they implement or by implementation issues from long ago. Here are some examples:

  • RADIANCE RGBE/XYZE (HDR): Dating from beginnings in 1985, this format is widely used. It uses a shared exponent for each texel, which works well for many images but not in general. Precision is limited and alpha is not supported. It is rather widely implemented, but a lack of documentation has made these implementations, even professional ones, variously broken.
  • OpenEXR: Powerful but complex. Implemented as three codependent libraries.
  • TIFF Extensions LogLuv and Pixar: Not simple, but standard implementations exist. Suffer from precision issues.
  • Various Proprietary Formats: Proprietary, therefore worthless.
  • Some NetPBM Formats: Complex; aliased with other graphics formats.
  • PFM: Actually a knockoff format of NetPBM, this one is actually almost decent; it still mixes ascii and binary and inherits more complexity from NetPBM.
  • Others: I am not aware of any other widely-used HDR format not falling into one of these categories (or inside these categories but worth mentioning individually). If I have overlooked something truly important, do let me know.
A good overview of some of these formats can be found here.

The common thread here is complexity. A lot of this complexity derives from the goal of efficiency—often from trying to pack a color value into 32 bits. This is understandable; modern applications benefit from reduced space requirements. However, as a simple, robust, clear format, these other HDR file formats are not.

F32 fills this gap. The basic version is unapologetically wasteful of space (generally by a factor of four), but precision is maintained perfectly and implementation is kept next-to-trivial. I doubt most applications are going to notice the extra millisecond paging from disk, and most people aren't going to care about the extra space required since it is so comparably small.

A slightly more complicated version compresses the data with zlib. This actually gives surprisingly good results. For example, the Grace Cathedral environment map (which is a standard test image in HDR imaging, available from this website) is 18.0MB when stored in HDR RADIANCE format. In uncompressed F32 (RGB), it is 54MB. However, when converted to compressed F32, it is only 17.0MB. If the HDR RADIANCE file is compressed, it's 13.5MB (21% smaller than F32)—but keep in mind that F32 is encoding four times as many bytes per texel! Plus it's a lot simpler and more flexible. Obviously, results vary—the point is the sizes are close, but you get so much more for it.

Features provided by F32 that may distance it from other HDR formats include:

  • Simplicity: Easy to implement, high-speed loading. Loading uncompressed texels in parallel is easy too if that's your thing.
  • Lossless: Stores 32-bit floats exactly as they appear in a rendering system.
  • High-resolution: Supports images up to 232-1 on a side. Your computer might not.
  • Documentation: It exists! See below.
  • Platform Independent: Well-defined, simple, standard requirements.
  • Free: As in freedom and as in beer.

Documentation

At this time, no (complete) reference implementation is available. This is primarily because F32 is so simple that writing one would be more an exercise in API calls to whatever-representation-in-memory-your-images-are than anything else.

F32 files have the extension ".f32" iff they are uncompressed and ".f32z" iff they are compressed. A F32 file is entirely binary. It comprises a header, possibly more metadata, and then the texel data. All datatypes that are more than 1 byte long are in little endian format (most desktop architectures today).

The first step of processing the file is to load the header. In C++11, the header looks like:

class HeaderF32 final {
	public:
		uint32_t magic;
		uint32_t version;

		uint32_t width;
		uint32_t height;

		uint32_t channel_specifier;

		uint32_t additional_prologue;
		
		uint64_t data_length;
};
The header fields:
  • ".magic": A magic number. Must be either "0x00000F32" or "0x80000F32". If it is the first, then it is an ordinary F32 file. If it is the second, then it is a compressed F32 file.
  • ".version": The version of F32 this file is. There is currently only one version: "1".
  • ".width": Width of the image.
  • ".height": Height of the image.
  • ".channel_specifier": Specifies the interpretation of the data. Think of it like an enum (although I have tried to make the values correspond to something intuitive for the basic types supported by version 1). As of version 1, the following are defined:
    • 1: single channel (probably some kind of grayscale or luminance)
    • 3: triple channel (probably some kind of RGB)
    • 4: quadruple channel (probably some kind of RGBA)
    Implementations may define their own custom channel options/semantics in addition to these, but the highest bit of the specifier must be set. If you define such a format that you think would be useful, let me know and I may give it my blessing. Note: I said "probably" because F32 does not enforce any kind of meaning for the channels. In practice, a three channel file will be sRGB photometric color space values (since everyone uses this for just about everything), but there is no obligation. It is up to the generating and loading applications to know what their own data represents.
  • ".additional_prologue": Version 1 supports no additional information. Any extra metadata that a F32 file-from-the-future may contain will have its length (in bytes) listed here. Version 1 loaders will skip this extra data.
  • ".data_length": The length of the data block, in bytes. This could, in principle, be calculated by other means (for uncompressed files by a multiplication, for compressed files by reading the file's length), but it is easiest to just write the length explicitly here.

The second step is to load any additional metadata specified by ".additional_prologue". Since no additional metadata is defined for version 1, this should be a no-op. To future-proof your loader, though, you should just skip ahead that many bytes in the file.

The third and final step is to load the texels' data. The image is stored in left-to-right, top-to-bottom order in the file; that is, data is read across first, and then down. The logical format of each texel is (as of version 1) either a single float, three floats, or four floats, corresponding to the meaning given by ".channel_specifier" above. The floats are all stored as IEEE 754 floating point (the standard floating point), and again, in little endian format.

If the image is uncompressed, then the data block is exactly as described above. If the image is compressed, then the data block is as described above but first compressed with the zlib library's function "compress2(...)". Decompression is done with "uncompress(...)".

No other transformations on the data are implied by the specification. There is no implicit "unit" for each texel (like radiance or flux or Watts or something); implementations are free to choose their own interpretation, in the same manner as they do for e.g. ".png" files (although, since this is an HDR format, radiometric units are preferred). In practice, though, people often put sRGB instead. Implementations are free to do post processing on the data, or to abuse the existing channel options for their own different purposes (although it is probably better to instead define a new value for ".channel_specifier" as stated above).

Example Data

Example F32 files are provided: (F32.f32), (F32z.f32z). These are HDR encodings of a standard-dynamic-range-image of a rose, which I believe I first saw on Wikipedia, but seems to originate perhaps with ImageMagick. If you know, let me know! The image's [0,255] range has been remapped to [0.0,1.0], and stores no alpha channel (so ".channel_specifier"==3). I'm pretty sure the values in the original image (and hence in this version) represent sRGB values. The image looks like:

I do not guarantee these images are correct, although it's probable they are. Let me know if you have trouble loading them.

Future work

Nothing planned.

Questions and FAQ

  • Does F32 support/define/obligate arbitrary color spaces?

    No. Color spaces really only exist to support photometry, and are a huge, incompatible mess that screws everything up. Consequently, in the name of simplicity, F32 refuses to define anything to do with them. F32 is a container format for raw data; it is up to the applications generating/using these files to know what they mean.

    In practice people either don't know or don't care about the distinction between radiometry and colorimetry/photometry, so they'll store nonradiometric values (most commonly, sRGB).

  • What does "F32" refer to?
    "F32" is used to refer broadly to the class of file formats discussed above. The file extensions ".f32" and ".f32z" (lowercase, as all files should be) are the required file extensions for the ordinary and the compressed F32 files, respectively. The name itself comes from storing IEEE 754 32-bit floating point values.
  • Do you support <whatever> metadata?
    No. Version 1 supports no additional metadata. This keeps the files small and simple. You're free to extend the format with ".additional_prologue" if you feel you must.
  • Does not compressing waste space?
    Maybe. Uncompressed F32 gets a lot for not being compressed: losslessness and simplicity are the two biggest advantages. The real question is: do you care? A 1024x1024 RGB F32 image takes up ~12MB. But, there exist some HDR images that need that much space. In any case, you're going to be using the HDR image anyway. If you can afford X amount of space in RAM, then you can certainly afford X amount of space on disk.
  • Does not using compression hurt loading time?
    Maybe. Modern operating systems load data from disk through DMA from the disk's cache, within which smaller F32 files will fit. Since most modern architectures are interrupt-driven, you'll probably not lose too much CPU time idling. If you really care about those extra milliseconds, you'd be doing this loading in parallel anyway. Keep in mind that, being direct, no per-pixel processing is required for uncompressed F32, so this helps it make up for any lost time.
  • What encoding does F32 use?
    Basic F32 is essentially just a direct mapping of the contents of an HDR image in memory plus a header tacked on the front. The translation is direct and lossless (relative to a RGBA float32 image).

Please contact me with any other questions you may have about F32, so that I may address them here. Especially, if you run into any implementation ambiguity, I want to resolve it. If there are any difficulties in implementation related to the specification, let me know that I may improve the wording.


COMMENTS
Ian Mallett - Contact -
Donate
- 2018 - Creative Commons License