Order Book Backtest Fills Don't Match? Debugging NautilusTrader batch_deltas and F_LAST
If you replay L2/L3 order-book data through NautilusTrader’s OrderBookDelta and the fills or position counts change after an upgrade, don’t immediately retune the strategy. Two different mechanisms deserve investigation: batch_deltas determines whether matching can see an intermediate order-book state; and cutting off the start of an event can go undetected even if its last row still carries F_LAST.
That second case is deceptively easy to miss. F_LAST identifies the end of a group; it does not prove that the first row of that group was present. A time window or filter_expr can remove an initial ADD while leaving the final DELETE marked F_LAST. The batcher may close what is now only a fragment. The version-pinned backtesting documentation explicitly warns about this failure to detect a truncated beginning.
I inspected the source and upstream tests for NautilusTrader v2.0.0rc7 and independently reproduced the event-boundary behavior using Python’s standard library. Use the configuration comparison below on your own Catalog; the appendix scripts illustrate the grouping mechanism with synthetic data rather than running the NautilusTrader engine.
Before changing the strategy: a four-step reconciliation checklist
This is the order in which I would investigate a backtest that changed despite apparently using the same data Catalog.
- Freeze the baseline. Record both framework versions, Catalog contents, time ranges, instrument precision, strategy version, and order/matching parameters. Do not regenerate the underlying historical data while investigating.
- Inspect event completeness. For each
instrument_id, checkF_LASTand especially whether the first event in the selected window was cut in half. A valid closing marker says nothing about the missing head. - Change one setting only. Run the same strategy, dataset and matching assumptions with
batch_deltas=Trueandbatch_deltas=False. Locate the first fill or callback difference, then inspect its exact deltas. - Decide on the baseline afterward. Choose the replay granularity according to the data provider’s event semantics, not according to which curve shows a higher return.
Adapt the documented BacktestDataConfig pattern in your existing backtest project. First configure batch_deltas=True, then change only that parameter to False for comparison; keep the real Catalog path and instrument_id identical between runs:
data = BacktestDataConfig(
data_type=NautilusDataType.OrderBookDelta,
catalog_path="/data/catalog",
instrument_id=instrument_id,
batch_deltas=True, # rc7 默认:按完整事件回放
)
# 对照组保持其余配置不变,仅将 batch_deltas 改为 False
The pinned Rust configuration sets the rc7 default to true. The Python configuration tests cover default, enabled, and disabled values. Setting False is a diagnostic control, not a repair for an incomplete historical book.
F_LAST can detect a missing ending, not necessarily a missing beginning
Suppose one logical market-data event adds an ask at 999.95 and then deletes it. Only the final DELETE carries F_LAST.
| Read window | Delta rows retained | What the event batcher can observe |
|---|---|---|
| Complete | ADD 999.95 → DELETE 999.95 (F_LAST) | Two rows grouped into a complete event |
| Tail truncated | ADD 999.95 without F_LAST | Pending group remains; an end-of-input error can be raised |
| Head truncated | DELETE 999.95 (F_LAST) only | The one-row fragment can still be closed |
Why does the last row pass through the grouping stage? The BookDeltasBatcher implementation maintains pending deltas by instrument, emits OrderBookDeltas when a row carries F_LAST, and checks for unclosed groups at the end of the input. It does not have enough information to reconstruct an ADD row already removed by the query.
“Can close” here refers specifically to the F_LAST grouping check; an invalid DELETE or a missing snapshot might fail a different downstream check. The practical fix is to verify event boundaries in the original Catalog or data feed, widening the window or starting from a trusted snapshot as needed. Counting how many rows contain F_LAST is not an adequate integrity check.
Why identical order-book deltas can produce different fills
The replay input may be complete and identical, yet the timing of matching still changes the outcome. Imagine a resting limit BUY at 1000.00 when the best ask is 1000.10. One event then adds an ask at 999.95 and removes it immediately; the DELETE carries F_LAST.
If the engine attempts matching after each individual delta, the buy can encounter the temporary 999.95 ask. If it applies the whole event first, that ask is gone by the time matching runs. This is a visibility-of-intermediate-state issue, not merely a batching performance optimization.
| Replay mode | Book state at the matching point | Synthetic toy fills |
|---|---|---|
| Individual deltas | Temporary 999.95 ask is visible | 1 |
Entire F_LAST event | 999.95 ask has already been removed | 0 |
The upstream Rust integration test builds a comparable temporary-ask situation and declares #[case::batched(true, 0)] and #[case::individual(false, 1)] as expected total_positions values. That metric comes from upstream test assertions. The 1/0 in the table is an independently executed toy model’s simulated fill count. Both clarify the same event-boundary mechanism, but they are not the same measurement.
The backtesting documentation also describes replaying the batch at the closing delta’s ts_init and changes to on_book_deltas callbacks and streaming chunk_size accounting. To debug a discrepancy, examine the earliest divergent event and callback rather than comparing only final PnL.
The v2.0.0rc6 configuration source lacks the batch_deltas field. This establishes a configuration-interface change; it does not prove that every old replay path was literally equivalent to False.
Reproducing the mechanism: full scripts and negative controls
I used Python 3.13.5’s standard library to model a deliberately simple book and matching step. The individual replay yields individual=1 and event-batched replay yields batched=0. I also checked missing trailing F_LAST, interleaved instruments, and separate events sharing one timestamp. The initial script passed 5/5 checks; an expanded unittest suite passed 10/10.
The most useful negative control is the case where the beginning of an event is missing but the remaining fragment still closes. A green result on that test proves the gap in F_LAST validation; it does not mean the damaged feed was recovered. Some checks overlap across the two suites, so fifteen passing assertions should not be advertised as fifteen independent market observations.
The complete runnable code and original output are preserved below. Save the first block as book_batching_fixture.py and run python3 book_batching_fixture.py. Place the second file, edge_case_verification.py, beside it and run python3 -m unittest -v edge_case_verification. Both rely only on the standard library and fictional prices: no queue-position logic, fees, historical feed or real exchange fills.
Original synthetic fixture — 5/5: full code and recorded output
"""Self-authored toy model of F_LAST event boundaries, NOT NautilusTrader runtime."""
from dataclasses import dataclass
from decimal import Decimal
@dataclass(frozen=True)
class Delta:
instrument: str
action: str
price: Decimal
last: bool
ts_init: int
def d(instrument, action, price, last, timestamp):
return Delta(instrument, action, Decimal(price), last, timestamp)
def group_by_last(rows):
pending = {}
for row in rows:
event = pending.setdefault(row.instrument, [])
event.append(row)
if row.last:
yield tuple(event)
del pending[row.instrument]
if pending:
detail = ", ".join(sorted(pending))
raise ValueError(f"missing F_LAST for {detail}")
def toy_fills(rows, batch_deltas):
"""One pending BUY at 1000; match after each replay item, not each event row."""
asks = set()
buy_pending = True
fills = 0
replay_items = group_by_last(rows) if batch_deltas else ((r,) for r in rows)
for item in replay_items:
for row in item:
if row.action == "ADD":
asks.add(row.price)
elif row.action == "DELETE":
asks.discard(row.price)
if buy_pending and asks and min(asks) <= Decimal("1000.00"):
buy_pending = False
fills += 1
return fills
def run():
# The same synthetic event shape as the concept behind the pinned upstream test.
rows = [
d("SIM", "ADD", "1000.10", True, 1),
d("SIM", "ADD", "999.95", False, 2),
d("SIM", "DELETE", "999.95", True, 2),
]
groups = list(group_by_last(rows))
assert [len(g) for g in groups] == [1, 2]
print("PASS 1/5 event boundaries: 3 deltas -> groups [1, 2]")
individual, batched = toy_fills(rows, False), toy_fills(rows, True)
assert (individual, batched) == (1, 0)
print(f"PASS 2/5 toy fills: individual={individual}, batched={batched}")
try:
list(group_by_last([d("SIM", "ADD", "1.00", False, 3)]))
except ValueError as exc:
assert "missing F_LAST for SIM" in str(exc)
else:
raise AssertionError("unterminated event did not fail")
print("PASS 3/5 unterminated event rejected")
interleaved = [
d("A", "ADD", "1", False, 4),
d("B", "ADD", "2", True, 4),
d("A", "DELETE", "1", True, 5),
]
received = list(group_by_last(interleaved))
assert [[r.instrument for r in group] for group in received] == [["B"], ["A", "A"]]
print("PASS 4/5 interleaved instruments remain isolated")
same_time = [d("SIM", "ADD", "1", True, 6), d("SIM", "ADD", "2", True, 6)]
assert [len(g) for g in group_by_last(same_time)] == [1, 1]
print("PASS 5/5 equal-timestamp completed events stay separate")
print("SUMMARY: 5/5 PASS, self-authored stdlib model only; no upstream engine run")
if __name__ == "__main__":
run()
PASS 1/5 event boundaries: 3 deltas -> groups [1, 2]
PASS 2/5 toy fills: individual=1, batched=0
PASS 3/5 unterminated event rejected
PASS 4/5 interleaved instruments remain isolated
PASS 5/5 equal-timestamp completed events stay separate
SUMMARY: 5/5 PASS, self-authored stdlib model only; no upstream engine run
Extended negative controls — 10/10: full code and recorded output
"""Independent negative-control suite for the self-authored F_LAST toy model.
Not NautilusTrader tests and not an exchange/market backtest.
"""
import unittest
from book_batching_fixture import d, group_by_last, toy_fills
class OrderBookEventBoundaryControls(unittest.TestCase):
def test_01_trailing_incomplete_event_is_rejected(self):
rows = [d("SIM", "ADD", "999.95", False, 10)]
with self.assertRaisesRegex(ValueError, "missing F_LAST for SIM"):
list(group_by_last(rows))
def test_02_head_truncated_event_is_not_detected_by_f_last(self):
full_event = [
d("SIM", "ADD", "999.95", False, 10),
d("SIM", "DELETE", "999.95", True, 10),
]
complete = list(group_by_last(full_event))
filtered = list(group_by_last(full_event[1:]))
self.assertEqual([len(g) for g in complete], [2])
# This is a NEGATIVE result: F_LAST alone cannot validate the missing head.
self.assertEqual([len(g) for g in filtered], [1])
self.assertEqual(filtered[0][0].action, "DELETE")
def test_03_unbatched_mode_accepts_unclosed_intermediate_row(self):
rows = [d("SIM", "ADD", "999.95", False, 10)]
self.assertEqual(toy_fills(rows, batch_deltas=False), 1)
with self.assertRaises(ValueError):
toy_fills(rows, batch_deltas=True)
def test_04_persistent_ask_matches_in_both_modes(self):
rows = [d("SIM", "ADD", "999.95", True, 10)]
self.assertEqual(toy_fills(rows, False), 1)
self.assertEqual(toy_fills(rows, True), 1)
def test_05_transient_ask_exposes_event_granularity(self):
rows = [
d("SIM", "ADD", "1000.10", True, 10),
d("SIM", "ADD", "999.95", False, 20),
d("SIM", "DELETE", "999.95", True, 20),
]
self.assertEqual((toy_fills(rows, False), toy_fills(rows, True)), (1, 0))
def test_06_equal_timestamps_do_not_merge_closed_events(self):
rows = [d("SIM", "ADD", "1", True, 10), d("SIM", "ADD", "2", True, 10)]
self.assertEqual([len(g) for g in group_by_last(rows)], [1, 1])
def test_07_interleaved_symbols_keep_independent_buffers(self):
rows = [
d("A", "ADD", "1", False, 10),
d("B", "ADD", "2", True, 10),
d("A", "DELETE", "1", True, 11),
]
self.assertEqual([[r.instrument for r in g] for g in group_by_last(rows)], [["B"], ["A", "A"]])
def test_08_partial_symbol_does_not_disappear_at_end(self):
rows = [d("A", "ADD", "1", False, 10), d("B", "ADD", "2", True, 11)]
iterator = iter(group_by_last(rows))
self.assertEqual([x.instrument for x in next(iterator)], ["B"])
with self.assertRaisesRegex(ValueError, "missing F_LAST for A"):
next(iterator)
def test_09_empty_input_is_accepted(self):
self.assertEqual(list(group_by_last([])), [])
def test_10_closing_delta_has_last_timestamp_in_toy(self):
rows = [d("SIM", "ADD", "1", False, 10), d("SIM", "DELETE", "1", True, 12)]
grouped = list(group_by_last(rows))
self.assertEqual(grouped[0][-1].ts_init, 12)
self.assertEqual(len(grouped[0]), 2)
if __name__ == "__main__":
unittest.main(verbosity=2)
test_01_trailing_incomplete_event_is_rejected (edge_case_verification.OrderBookEventBoundaryControls.test_01_trailing_incomplete_event_is_rejected) ... ok
test_02_head_truncated_event_is_not_detected_by_f_last (edge_case_verification.OrderBookEventBoundaryControls.test_02_head_truncated_event_is_not_detected_by_f_last) ... ok
test_03_unbatched_mode_accepts_unclosed_intermediate_row (edge_case_verification.OrderBookEventBoundaryControls.test_03_unbatched_mode_accepts_unclosed_intermediate_row) ... ok
test_04_persistent_ask_matches_in_both_modes (edge_case_verification.OrderBookEventBoundaryControls.test_04_persistent_ask_matches_in_both_modes) ... ok
test_05_transient_ask_exposes_event_granularity (edge_case_verification.OrderBookEventBoundaryControls.test_05_transient_ask_exposes_event_granularity) ... ok
test_06_equal_timestamps_do_not_merge_closed_events (edge_case_verification.OrderBookEventBoundaryControls.test_06_equal_timestamps_do_not_merge_closed_events) ... ok
test_07_interleaved_symbols_keep_independent_buffers (edge_case_verification.OrderBookEventBoundaryControls.test_07_interleaved_symbols_keep_independent_buffers) ... ok
test_08_partial_symbol_does_not_disappear_at_end (edge_case_verification.OrderBookEventBoundaryControls.test_08_partial_symbol_does_not_disappear_at_end) ... ok
test_09_empty_input_is_accepted (edge_case_verification.OrderBookEventBoundaryControls.test_09_empty_input_is_accepted) ... ok
test_10_closing_delta_has_last_timestamp_in_toy (edge_case_verification.OrderBookEventBoundaryControls.test_10_closing_delta_has_last_timestamp_in_toy) ... ok
----------------------------------------------------------------------
Ran 10 tests in 0.001s
OK
Should the new results replace the old backtest baseline?
Fewer fills under batch_deltas=True do not, by themselves, mean “more realistic profit,” and setting False merely to preserve an attractive equity curve is no better. First establish the provider’s event boundaries; only then interpret the fill differences. If several deltas form one logical event, consuming them atomically respects that convention. If the window is missing its beginning, neither boolean value restores the lost state.
Keep the previous results, locate the first diverging OrderBookDelta, and inspect its F_LAST, ts_init, and original grouping before analyzing fees, queue assumptions, trading costs or strategy returns. The same underlying lesson appears in the vectorbt CVaR sample-denominator case and repeated-stock-selection win-rate denominators: a quantitative number is only interpretable after defining what counts as one complete observation.
Version and verification note: Sources are pinned to NautilusTrader v2.0.0rc7, commit 55c0e82053142bce3a5cba4de4221d3929c2d3bc. The Python 3.13.5 synthetic fixture and 5+10 boundary checks were recorded on October 18, 2026. This is a teaching example, not an upstream NautilusTrader run or a historical/live trading backtest. It covers replay engineering, not investment advice.