histogramQuantile() function

histogramQuantile() approximates a quantile given a histogram that approximates the cumulative distribution of the dataset.

Each input table represents a single histogram. The histogram tables must have two columns – a count column and an upper bound column.

The count is the number of values that are less than or equal to the upper bound value. The table can have any number of records, each representing a bin in the histogram. The counts must be monotonically increasing when sorted by upper bound. If any values in the count column or upper bound column are null, it returns an error. The count and upper bound columns must not be part of the group key.

The quantile is computed using linear interpolation between the two closest bounds. If either of the bounds used in interpolation are infinite, the other finite bound is used and no interpolation is performed.

Output tables

Output tables have the same group key as corresponding input tables. Columns not part of the group key are dropped. A single value column of type float is added. The value column represents the value of the desired quantile from the histogram.

Function type signature

(
    <-tables: stream[A],
    ?countColumn: string,
    ?minValue: float,
    ?onNonmonotonic: string,
    ?quantile: float,
    ?upperBoundColumn: string,
    ?valueColumn: string,
) => stream[B] where A: Record, B: Record

Parameters

quantile

Quantile to compute. Value must be between 0 and 1.

countColumn

Column containing histogram bin counts. Default is _value.

upperBoundColumn

Column containing histogram bin upper bounds. Default is le.

valueColumn

Column to store the computed quantile in. Default is `_value.

minValue

Assumed minimum value of the dataset. Default is 0.0.

onNonmonotonic

Describes behavior when counts are not monotonically increasing when sorted by upper bound. Default is error.

Supported values:

• error: Produce an error.
• force: Force bin counts to be monotonic by adding to each bin such that it is equal to the next smaller bin.
• drop: When a nonmonotonic table is encountered, produce no output. If the quantile falls below the lowest upper bound, interpolation is performed between minValue and the lowest upper bound. When minValue is equal to negative infinity, the lowest upper bound is used.

tables

Input data. Default is piped-forward data (<-).

Examples

Compute the 90th quantile of a histogram

data
    |> histogramQuantile(quantile: 0.9)