/ˈprɛ-ʒɪ-ʒən/

noun — “how finely a system can distinguish truth from its nearest approximation.”

Precision is the measure of how detailed or exact a numerical representation or computation can be. In computing, it describes the degree to which a value can be represented without losing information due to rounding, truncation, or storage limits. It is closely tied to floating-point systems such as IEEE 754.

At a simple level, precision answers the question: how many meaningful digits can a system reliably store and compute?

It is important to separate two ideas that often get blended together:

  • Precision — how detailed a representation is
  • Accuracy — how close a value is to the true or intended value

A system can be highly precise but still inaccurate, or accurate but not very precise. Computing often deals with both constraints simultaneously, especially when representing real numbers in binary form.

// precision vs accuracy (conceptual)

value: 1.0000000000000001   ← high precision, possibly inaccurate
value: 1.0                  ← lower precision, but may be closer to truth

In digital systems, precision is limited by the number of bits used to store a value. For example, a 32-bit floating-point number has fewer representable digits than a 64-bit floating-point number.

// IEEE 754 precision levels

32-bit float  → ~7 decimal digits of precision
64-bit float  → ~15–16 decimal digits of precision

This limitation means that many real-world values must be approximated. As numbers grow or calculations repeat, small errors accumulate, linking precision directly to phenomena such as rounding error.

Precision is not just about storage, but also about computation. Operations like addition, subtraction, and multiplication can amplify small representation errors depending on how numbers are structured.

For example:

// precision loss through subtraction

a = 1.0000001
b = 1.0000000
result = a - b
// result is small, but sensitive to rounding behavior

In systems like graphics engines or physics simulations, this sensitivity can determine whether objects behave smoothly or exhibit subtle instability over time. In financial systems, it can determine whether values remain consistent across repeated calculations.

Precision is also closely tied to the concept of numeric range. Increasing precision often reduces range unless more bits are allocated, which is why formats like IEEE 754 carefully balance exponent size (range) and significand size (precision).

// trade-off in floating-point design

more precision → more bits for significand
more range     → more bits for exponent
fixed size     → unavoidable trade-off

Outside of floating-point arithmetic, precision appears in many other contexts:

  • Integer precision — exact representation of whole numbers within a fixed bit-width
  • Decimal precision — number of digits shown or stored in base-10 systems
  • Measurement precision — resolution of physical or experimental instruments

In all cases, precision defines a boundary: how finely a system can slice continuity into discrete representable parts. Beyond that boundary, values either collapse into approximation or require more storage than the system can provide.

Conceptually, precision is the “grain size” of computation. Fine grain allows subtle distinctions but requires more space and processing. Coarse grain is efficient but hides detail. Every system chooses a balance between the two, often without the user ever seeing the trade-off directly.

It also explains why computers sometimes behave in ways that feel slightly off when dealing with real numbers. They are not wrong so much as constrained: operating within a fixed precision budget that cannot fully capture the continuum of real mathematics.

Ultimately, precision is less about correctness and more about resolution. It defines how sharply a system can see the numerical world it is simulating.

See float, IEEE 754, rounding error, decimal, binary

Computing
Concept
Granularity

See More