Getting Started with bitfield

This guide walks through the core workflow of the bitfield package: planning an encoding, choosing protocols, finding optimal bit allocations, and sharing protocols with the community.

The example data

The package ships with bf_tbl, a small dataset with typical quality issues (missing values, invalid coordinates, mixed formats):

bf_tbl
#> # A tibble: 9 × 5
#>       x     y commodity yield year 
#>   <dbl> <dbl> <fct>     <dbl> <chr>
#> 1  25.3  59.5 soybean   11.2  2021 
#> 2  27.9  58.1 maize     12.0  <NA> 
#> 3  27.8  57.8 soybean   13.2  2021r
#> 4  27    59.2 <NA>       4.43 2021 
#> 5 259   Inf   honey     13.0  2021 
#> 6  27.3  59.1 maize      8.55 2021 
#> 7  26.1  58.4 soybean   11.3  2021 
#> 8  26.5 NaN   maize     10.6  2021 
#> 9   0     0   soybean    9.01 2021

Encoding types at a glance

Every bit flag uses one of a few encoding types. The table below summarises when to use which:

Encoding	Protocol	Key parameters	Use when
Boolean	`na`, `inf`, `matches`	`set`	Yes/no flags (1 bit each)
Enumeration	`category`	`na.val`	Discrete classes (auto-sized)
Integer	`integer`	`range`, `fields`	Bounded values with uniform precision
Floating-point	`numeric`	`fields` (exp/sig)	Open-ended values spanning orders of magnitude

Integer encoding maps a bounded range linearly onto bit states. Good for percentages, indices, and quantities with well-defined limits.

Floating-point encoding splits bits into exponent and significand fields. Precision is finest near zero and coarsens with magnitude. Good for standard deviations, rates, and other variables that can span several orders of magnitude.

When in doubt, bf_analyze() helps decide.

Using bf_analyze() to find optimal bit allocations

bf_analyze() works with any data type. For booleans, integers, and categories it reports the required bit count directly. For floating-point data it evaluates all possible exponent/significand configurations and reports those on the Pareto front for each total bit count.

set.seed(42)
x <- runif(1000, 0.1, 10)
bf_analyze(x, range = c(0, 15), max_bits = 8, decimals = 1)
#> Float Analysis
#> ==============
#> 
#>   Observations      1000
#>   NA values         0
#>   Range             [0.102365, 9.98506]
#>   Levels            -
#>   Sign required     no
#>   Bits required     select from the table below
#>   Suggested na.val  automatic 
#>   Target range      [0.102365, 15]
#>   Decimals          1
#> 
#> Exp  Sig  Total  Underflow  Overflow  Changed     Min Res     Max Res        RMSE     Max Err
#> ---  ---  -----  ---------  --------  -------  ----------  ----------  ----------  ----------
#>   2    1      3      4.2%    19.8%    95.8%      0.2500      2.0000      2.3762      6.0000
#>   2    2      4      4.2%    19.8%    91.9%      0.1250      1.0000      1.4195      4.0000
#>   3    1      4      0.7%     0.0%    94.0%      0.0625      4.0000      0.9339      2.0000
#>   2    3      5      4.2%    19.8%    84.5%      0.0625      0.5000      0.9514      3.0000
#>   3    2      5      0.7%     0.0%    89.6%      0.0312      2.0000      0.6466      2.0000
#>   4    1      5      0.0%     0.0%    93.3%      0.0312      4.0000      0.9338      2.0000
#>   2    4      6      4.2%    19.8%    70.6%      0.0312      0.2500      0.7265      2.5000
#>   3    3      6      0.7%     0.0%    80.7%      0.0156      1.0000      0.3072      1.0000
#>   4    2      6      0.0%     0.0%    88.9%      0.0156      2.0000      0.6465      2.0000
#>   2    5      7      4.2%    19.8%    56.1%      0.0156      0.1250      0.5992      2.2000
#>   3    4      7      0.7%     0.0%    65.3%      0.0078      0.5000      0.1626      0.5000
#>   4    3      7      0.0%     0.0%    80.0%      0.0078      1.0000      0.3071      1.0000
#>   2    6      8      4.2%    19.8%    40.7%      0.0078      0.0625      0.5578      2.1000
#>   3    5      8      0.7%     0.0%    48.7%      0.0039      0.2500      0.0895      0.3000
#>   4    4      8      0.0%     0.0%    64.6%      0.0039      0.5000      0.1624      0.5000
#> 
#> Usage:
#>   bf_map(protocol = "numeric", ...,
#>          fields = list(exponent = <exp>, significand = <sig>))

The Pareto table shows multiple rows per bit count when trade-offs exist. Key columns:

Underflow/Overflow – percentage of values outside the representable range
Changed – percentage differing after round-trip encoding at the specified decimal precision
Min/Max Res – step size at smallest and largest representable values
RMSE / Max Err – encoding error metrics

For example, at 7 total bits, exp=4/sig=3 covers the full range (no underflow) while exp=3/sig=4 offers finer precision but cannot represent values below \(2^{-3} = 0.125\). Neither dominates the other, so both appear.

Use the result to pick a configuration, then pass it to bf_map():

# After deciding on exp=4, sig=3 based on bf_analyze() output:
reg <- bf_map(protocol = "numeric", data = my_data, x = sd_values,
              registry = reg, fields = list(exponent = 4, significand = 3))

A complete example

Here is a full workflow using bf_tbl:

# 1. Create a registry
reg <- bf_registry(
  name = "yield_QA",
  description = "Quality assessment for yield data",
  template = bf_tbl)

# 2. Add boolean flags
reg <- bf_map(protocol = "na", data = bf_tbl, x = commodity, registry = reg)
reg <- bf_map(protocol = "inf", data = bf_tbl, x = x, registry = reg)
reg <- bf_map(protocol = "inf", data = bf_tbl, x = y, registry = reg)

# 3. Add a category
reg <- bf_map(protocol = "category", data = bf_tbl, x = commodity,
              registry = reg, na.val = 0L)

# 4. Add a numeric value with custom float encoding
reg <- bf_map(protocol = "numeric", data = bf_tbl, x = yield,
              registry = reg, format = "half")

reg
#>   type  data.frame
#>   width 21
#>   flags 5  -|-|-|--|----------------
#> 
#>   pos encoding  name     col
#>   1   0.0.1/0   na       commodity
#>   2   0.0.1/0   inf      x
#>   3   0.0.1/0   inf      y
#>   4   0.0.2/0   category commodity
#>   6   1.5.10/15 numeric  yield

# 5. Encode and decode
field <- bf_encode(registry = reg)
decoded <- bf_decode(field, registry = reg, verbose = FALSE)

head(decoded, 3)
#> $na_commodity
#> [1] 0 0 0 1 0 0 0 0 0
#> 
#> $inf_x
#> [1] 0 0 0 0 0 0 0 0 0
#> 
#> $inf_y
#> [1] 0 0 0 0 1 0 0 0 0

Verify round-trip integrity

Always check that the encode/decode cycle preserves essential information:

# input NAs
sum(is.na(bf_tbl$commodity))
#> [1] 1

# NAs after roundtrip
sum(decoded$na_commodity == 1)
#> [1] 1

Design guidelines

Match precision to needs. Do not allocate 16 bits where 5 suffice. Use bf_analyze() to quantify the trade-off.

Use atomic protocols. Each bf_map() call should test one concept. This maximises reusability across projects.

Handle NA explicitly. Use the na.val parameter to reserve a sentinel value for missing data. Choose a value outside your data range (can be 0 when that value is not relevant in and of itself, or the maximum integer state).

Test with edge cases. Include NA, Inf, NaN, zeros, and boundary values in your test data:

problematic <- bf_tbl[c(4, 5, 8, 9), ]
print(problematic)
#> # A tibble: 4 × 5
#>       x     y commodity yield year 
#>   <dbl> <dbl> <fct>     <dbl> <chr>
#> 1  27    59.2 <NA>       4.43 2021 
#> 2 259   Inf   honey     13.0  2021 
#> 3  26.5 NaN   maize     10.6  2021 
#> 4   0     0   soybean    9.01 2021

Choose units wisely. Livestock density in animals/km\(^2\) (range 0–290) needs 9 bits. The same quantity in animals/ha (range 0–3.1) needs only 5 bits. Unit choice directly affects bit budget.

Creating custom protocols

When built-in protocols do not cover your use case, create a custom one with bf_protocol(). For example, some datasets embed status flags directly in value strings – a year column might contain "2021r" where r marks the value as revised. The built-in grepl protocol can detect such flags as a boolean, but a custom protocol can distinguish which flag is present:

valueFlag <- bf_protocol(
  name = "valueFlag",
  description = paste("Extracts trailing status flags from {x}.",
                      "0 = none, 1 = r(evised), 2 = p(rovisional),",
                      "3 = e(stimated)"),
  test = "function(x) { suffix <- sub('.*([a-z])$', '\\\\1', x); match(suffix, c('r','p','e'), nomatch = 0L) }",
  example = list(x = c("2020", "2021r", "2019p", "2018e", NA)),
  type = "int",
  bits = 2
)

Versioning and extension

When improving existing protocols, use versioning to maintain reproducibility:

valueFlagV2 <- bf_protocol(
  name = "valueFlag",
  description = paste("Extracts trailing status flags from {x}.",
                      "0 = none, 1 = r(evised), 2 = p(rovisional),",
                      "3 = e(stimated). Now also handles uppercase flags."),
  test = "function(x) { suffix <- sub('.*([a-zA-Z])$', '\\\\1', tolower(x)); match(suffix, c('r','p','e'), nomatch = 0L) }",
  example = list(x = c("2020", "2021r", "2019P", "2018e", NA)),
  type = "int",
  bits = 2,
  version = "1.1.0",
  extends = "valueFlag_1.0.0",
  note = "Now handles uppercase flags via case-insensitive matching"
)

Sharing protocols via community standards

The bitfloat/standards repository enables sharing encoding protocols. Access it through bf_standards():

# List available protocols
bf_standards(action = "list")

# Pull a community protocol
soil_protocol <- bf_standards(
  protocol = "soil_moisture",
  remote = "environmental/soil",
  action = "pull"
)

# Push your own protocol
bf_standards(
  protocol = dataAgeProtocol,
  remote = "environmental/temporal",
  action = "push",
  version = "1.0.0",
  change = "Initial release: data age encoding for environmental monitoring"
)

This requires a GitHub Personal Access Token. See ?bf_standards for setup instructions.

Quick reference

Do:

Plan your encoding before writing code
Use bf_analyze() for floating-point configurations
Test encode/decode round-trips
Use na.val for missing data
Keep protocols atomic (one concept each)
Document registries with descriptive names

Avoid:

Over-allocating bits for unnecessary precision
Encoding the same concept across multiple flags
Skipping edge case testing
Using floating-point encoding for bounded, uniform-precision variables

For more details, see the function documentation (?bf_map, ?bf_analyze, ?bf_protocol) and the package website at https://bitfloat.github.io/bitfield/.