Hey, I’ve been mapping out a fault‑tolerant logging pipeline that could run on a single Raspberry Pi cluster. It uses a custom LSM‑tree for writes and a checksum‑verified append‑only store. Interested in how that stacks against your usual hash‑based integrity checks?

Exactly. I’m still playing with a rolling checksum per block so the overhead stays under 1 % of the raw data. The trick is to store the checksum only once in the index, not with every write. That keeps the write amplification low while giving me a full‑block sanity check. If anything goes wrong I can pinpoint the exact corrupted sector. If you’re running low on space, just tweak the block size; larger blocks mean fewer checksums, but the trade‑off is less granularity when something does break. Happy to share the script if you want to run a quick audit on your own cluster.

Here’s a quick Bash sketch that splits a file into 1 MiB blocks, computes a 64‑bit CRC32C (bigger than plain CRC32, still fast), and writes the checksums to a side file. Adjust `BLOCK_SIZE` to see the trade‑off. ```bash #!/usr/bin/env bash # audit.sh – block‑level checksum audit for a Pi cluster # Usage: ./audit.sh /path/to/data/file /tmp/checksums.txt set -euo pipefail DATAFILE="$1" OUTFILE="$2" BLOCK_SIZE=${BLOCK_SIZE:-1048576} # 1 MiB # Ensure the checksum tool exists (on Pi OS you can use crc32c from libgoogle-perftools-dev) command -v crc32c >/dev/null || { echo "crc32c missing – install libgoogle-perftools-dev"; exit 1; } # Remove old checksum file if it exists > "$OUTFILE" printf "Processing %s with %d‑byte blocks...\n" "$DATAFILE" "$BLOCK_SIZE" # Read the file block‑by‑block, compute CRC32C, and append to the output dd if="$DATAFILE" bs="$BLOCK_SIZE" count=0 2>/dev/null | \ while read -r -n "$BLOCK_SIZE" block; do # Compute checksum of the block chk=$(printf "%s" "$block" | crc32c) # Write checksum to file printf "%s\n" "$chk" >> "$OUTFILE" done echo "Done. Checksum file: $OUTFILE" ``` A few notes: * `crc32c` gives you a 64‑bit value that’s still very fast on ARM. If you need cryptographic strength, swap it for `sha256sum` – just remember the overhead spikes. * The script writes one checksum per block. If you bump `BLOCK_SIZE` to, say, 4 MiB, you’ll halve the number of entries, but you’ll lose granularity—an error inside that block will look like a larger corruption. * Write amplification: each block read for checksum is a read operation. In a write‑heavy system you can pre‑compute checksums during the write path and just update the index, which keeps the read overhead minimal. Run it on your Pi cluster, tweak `BLOCK_SIZE`, and compare how the checksum file size and verification time change. Happy auditing!