ByteFlow is a next-byte predictor trained directly on raw PCAP flow bytes. It has no tokenizer, no protocol parser, and no notion of “field” or “record.” Its output is a probability distribution over the next byte, and anomaly detection falls out of how confident that distribution is: bits-per-byte (BPB). A model that has seen 366k benign HTTP flows compresses a new benign HTTP flow tightly. It has no such fluency with a C2 beacon, so it spends more bits per byte to encode it.
Pipeline
Input is a fixed window of T=2048 raw bytes. Five stages process it:
- LocalEncoder — 4 transformer layers, sliding-window causal attention with window=128, embedding dimension d=512. Each attention and FFN block is followed by Canon, a learned 4-tap causal convolution.
- CodingRateChunker — selects K=256 positions from the 2048 as chunk boundaries, using the L2² norm of the local encoder’s hidden state at each position as a proxy for information density (the ΔR term from coding rate theory). Position 0 is always included; the remaining 255 are the highest-norm positions in the sequence. Selected states are projected up to d=1024.
- GlobalTransformer — 6 layers of full causal attention over just the 256 selected positions, using RoPE and RMSNorm. This is where the quadratic attention cost lives, and restricting it to the chunker’s output rather than all 2048 bytes is what makes the context length tractable.
- Upsampler — broadcasts each global chunk’s representation back to every byte position it covers, using 16 learned per-bin projection matrices (W_bin) to adapt the global signal to a byte’s local position within its chunk. The result is added to the LocalEncoder’s original representation.
- LocalDecoder — 4 more sliding-window transformer layers, projecting to a 259-entry vocabulary (256 byte values, plus BOS/PAD, DIR_FLIP, and FLOW_SEP tokens) for the final next-byte logits.
The mid config used for the current checkpoint runs T=2048, K=256, 6 global layers, for 143.5M parameters total. A larger config (T=4096, K=512, 12 global layers) exists but is untrained.
Data
The benign training corpus is 825,686 flows (3.83 GB), sourced from the pcapultimate collection: Wireshark sample captures, ICS conference PCAPs, and public repositories. Protocol distribution is heavily skewed toward HTTP (44.3%) and DNS (37.7%), with SMTP at 10.3% and TLS at 7.5%. SMB and FTP have six flows each, effectively unrepresented. This means BPB scores on SMB and FTP traffic currently reflect the model never having learned those protocols’ byte statistics, not genuine anomaly detection.
Malware data, used only for evaluation probes and never for training, totals 616,810 flows across CTU-13 botnet captures (489k train-split flows) and elcabezzonn red-team/exploit captures (4k train-split flows), spanning dns, http, tls, smtp, smb, and ftp.
A balanced retraining corpus is planned at ~500k flows: HTTP and DNS downsampled to 150k each, SMTP held at 85k, TLS supplemented to 80k, and SMB/FTP built up to 30k+ and 10k+ respectively from CICIDS-2017 and other sources.
Training run and results
The first training run stopped at step 79,900 when Colab compute units ran out; the usable checkpoint is step 75,000. Mean BPB over the run: 1.8 at step 5k, 1.3 at 10k, 0.87 at 18k, 0.72 at 25k, 0.60 at 32k, 0.38 at 40k. At step 75k, the benign/botnet gap probe reported mean benign BPB of 0.34 against mean botnet BPB of 5.07, a gap of 4.73.
Threshold calibration
Current guidance, specific to the step-75k checkpoint on the imbalanced corpus: below 2.0 BPB is clean (benign p95 = 1.01), 2.0–4.0 BPB is suspicious and worth logging, above 4.0 BPB is high-confidence anomalous (botnet p25 = 3.37). Reliability varies by protocol. HTTP is strongest, with a benign median of 0.06 BPB, backed by 366k training flows. DNS is strong on 312k flows. SMTP is reliable. TLS is present but weaker, since the model can only score the unencrypted handshake and metadata, not the encrypted payload. SMB and FTP are unreliable until the balanced corpus retrain.
Inference
ByteFlowScorer loads a checkpoint and exposes score_flow/score_flows over raw bytes, returning BPB, byte counts, truncation status, and an is_anomaly flag against a configurable threshold. Flows longer than 2046 bytes are truncated to the model’s context window; sliding-window scoring across longer flows is not implemented. pcap_scan.py extracts and scores flows from a PCAP directly, and score_cli.py wraps this for command-line use, supporting both training-mode flow keying (min_port, max_port, transport) and 5-tuple keying for real mixed traffic.
Planned work
Balanced corpus retraining is in progress, with downloads pending for the additional SMB, FTP, and TLS sources. Beyond that: a web interface for PCAP upload and per-flow reporting, sidecar deployment alongside Zeek/Suricata for real-time scoring, environment-specific fine-tuning, and TensorRT optimization for production throughput.