btclib_node.chainstate package

Submodules

btclib_node.chainstate.block_index module

BlockIndex, every header this node has seen and which chain is active.

BlockStatus tracks a header from valid_header up through however far its block has been validated; get_download_candidates and MAX_DOWNLOAD_WINDOW are what bound how far ahead of the active chain a download is allowed to run, read from both download.py and here. invalidate is what a failed contextual check calls, through main.update_header_index, to drop a header and everything built on it. stage_status and finalize are set_status split into its two halves – the in-memory move and the disk write – so that main._finalize_fork can hold the second half back across more than one block; db.py’s own docstring is where that staging, shared with UtxoIndex, is argued.

set_downloaded and set_status both check pending before writing through, and for the same reason: either can be asked to change a hash pending already holds, unflushed, and a write-through there would only be undone the next time finalize writes that pending entry’s own stale snapshot back over it.

For set_status this is reached because invalidate’s own caller can name a hash that already connected once. update_chain sets failed_hash to a block across utxo_index.add_block, _validate_block, block_db.add_rev_block and filter_index.add_connected_block alike, so a fault in either of the last two – an I/O failure, nothing to do with the block’s own content – invalidates a block exactly as an actual validation failure would. Reached during a chain-tip flip-flop – a hash stage_status staged, disconnected by a later trial that re-stages it there, then offered again – that is a hash pending still holds, unflushed. set_status therefore checks pending itself: a hash already staged there is updated in pending, exactly as stage_status would leave it, rather than written straight through, so the next finalize writes the invalidation instead of clobbering it with the stale entry write-through would otherwise race against. btclib-org/btclib-node#586.

For set_downloaded this is reached from main.prune_up_to_height: _finalize_fork’s own to_add loop stages every hash a fork connects, through stage_status, before that fork’s own finalize ever runs – and to_add is not bounded by MIN_BLOCKS_TO_KEEP anywhere. get_fork_details walks back to the common ancestor with no depth limit, _ready_fork accepts whatever it returns once every hash is downloaded, and MAX_DOWNLOAD_WINDOW allows up to 1024 – so a single update_chain call connecting a fork longer than the retained depth stages hashes into pending that prune_up_to_height, run once at the end of that same call, reaches too. p2p.callbacks.block only ever sets the flag on a hash not yet downloaded, at or next to the tip, which is never one pending holds; prune_up_to_height is the caller this check is for.

class btclib_node.chainstate.block_index.BlockIndex(parent_db: KeyValueStore, chain: Chain, logger: Logger)

Bases: object

Every header this node has seen, and which chain among them is active.

header_dict maps a hash to its BlockInfo; chainwork holds each one’s cumulative work, kept apart from the record itself since it is derived rather than stored (issue #201). active_chain is the current best chain by hash; block_candidates is every other header that might still beat it once downloaded; header_index is the best known header chain, tracked separately since a header being known does not make its own block downloaded, let alone valid. header_index_pos is header_index’s own hash -> position, kept beside it the same way chainwork is kept beside header_dict (issue #439).

add_headers(headers: Iterable[__annotationlib_name_1__]) → bytes | None

Validate headers as one batch, then index every one of them.

Returns the highest header this batch carried that is indexed now (new or already known), or None if the batch connects to nothing this index knows at all.

add_to_active_chain(block_hash: bytes) → None

Append block_hash to active_chain, with no check it connects.

calculate_chainwork() → None

Compute every header’s cumulative work into chainwork.

Backfills children along the way, one entry per header visited.

finalize(wb: KeyValueStore | None = None) → None

Write every status stage_status staged, into wb if there is one.

Mirrors FilterIndex.finalize: a write_batch of its own when no wb is given, one write inside a caller’s own batch otherwise.

generate_active_chain() → None

Rebuild active_chain from every header marked in_active_chain.

generate_block_candidates() → None

Rebuild block_candidates from every valid_header past the tip.

generate_header_index() → None

Rebuild header_index, seeded from active_chain then extended.

get_block_info(block_hash: bytes) → BlockInfo

Return the BlockInfo stored for block_hash.

get_block_locator_hashes() → list[bytes]

Return a block locator over header_index, its own best known chain.

Exponentially sparser going back from its own tip, always including its genesis – the shape Core’s own LocatorEntries builds, cited in the comment below.

get_download_candidates() → list[bytes]

Return every undownloaded block a candidate branch still needs.

Walks each entry of block_candidates back from its own tip, collecting every not-yet-downloaded hash until it reaches one already seen or already on the active chain, then returns the union in height order, capped at MAX_DOWNLOAD_WINDOW.

get_first_candidate() → BlockInfo | None

Return the first downloaded candidate outweighing the active chain.

Pops every stale entry (work below the active chain’s own) off the front of block_candidates, then scans up to the 100 left: among those still ahead on work, the first whose whole branch is downloaded is returned, or the very first of them if none is, or None if there is no candidate ahead at all.

get_fork_details(header_hash: bytes, chain: list[bytes] | None = None) → tuple[list[bytes], list[bytes]]

Split chain at its common ancestor with header_hash.

chain defaults to active_chain. Returns the branch from that ancestor up to header_hash (ancestor excluded, oldest first) and the tail of chain that branch would replace.

get_headers_from_locators(block_locators: Sequence[__annotationlib_name_1__], stop: bytes) → list[BlockHeader]

Return up to 2000 headers after the first locator this index knows.

block_locators is read in the caller’s own order, so the first one found in header_index is where the answer resumes from. Stops at stop if reached first, and returns nothing if none of block_locators is known.

Membership and position both come from header_index_pos rather than a scan of header_index itself (btclib-org/btclib-node#439). The slice is capped at 2000 before stop is looked for, rather than after: stop is looked for inside the capped slice, not the whole of header_index, which is what btclib-org/btclib-node#434 raised ValueError on – a stop at or below block_locator’s own height is never in the slice taken after it, so it is simply not found rather than raising, and the answer is the slice unchanged: empty where the locator is already this index’s own tip, Core’s own “nothing to send” for the same request.

init_from_db() → None

Load every stored header into header_dict, then derive the rest.

Stops at the first key that is not a blkinfo- record: the shared store’s own key order (db.py’s docstring) sorts this index’s own keys ahead of the filter and UTXO indexes sharing the same store.

invalidate(block_hash: bytes) → None

Mark block_hash invalid, and everything indexed on top of it.

Walks children rather than header_dict, so the cost is the size of the bad lineage rather than of the whole index. Every invalidated hash is dropped from block_candidates; header_index is rebuilt from active_chain only if it held one of them.

remove_from_active_chain(block_hash: bytes) → None

Pop active_chain’s tip if it is block_hash, else raise.

ChainstateInconsistencyError, since a caller removing anything else is reorganizing the chain out of order.

set_downloaded(block_hash: bytes, *, downloaded: bool = True) → None

Set block_hash’s downloaded flag, replacing its BlockInfo.

Checks pending first, the same shape set_status above already does and for the same reason: a hash pending still holds would have a write-through here undone the next time finalize writes that pending entry’s own stale snapshot back over it. main.prune_up_to_height calling this on a hash _finalize_fork’s own to_add loop just staged, in the same update_chain call, before that fork’s own finalize ever runs, is exactly that: to_add is not bounded by MIN_BLOCKS_TO_KEEP anywhere, so a fork longer than the retained depth reaches pending at least as far back as prune_up_to_height’s own clearing range – Core has no counterpart to this race, since CBlockIndex flags are in-memory and m_dirty_blockindex is flushed whole rather than in halves.

set_status(block_hash: bytes, status: BlockStatus, wb: KeyValueStore | None = None) → None

Set block_hash’s own status, replacing its stored BlockInfo.

Writes straight through to the store (or to a caller’s own wb) unless pending already holds this hash – in which case the change is folded into that pending entry instead, exactly as stage_status below would leave it, and wb goes unused: the write is deferred to finalize’s own flush rather than happening now at all, since a write-through here would only be undone the next time finalize writes that pending entry’s stale value over it. The module docstring argues why this happens – invalidate’s own caller, update_chain.

stage_status(block_hash: bytes, status: BlockStatus) → None

Set block_hash’s status now, its write staged for finalize.

set_status above writes through to the store (or to a caller’s own wb) the moment it is called, unless pending already holds the hash; this stages the write into pending unconditionally, for finalize to write out whenever it next runs – _finalize_fork’s own to_add/to_remove loop is the one caller, once per block a fork connects or disconnects, so that a block’s own status reaches disk only together with the UTXO cache’s flush rather than one write_batch per block. A later call for the same hash before that flush – a reorg undoing a connection this process staged and never wrote – simply replaces the pending entry, which is correct: only the state finalize is about to write ever needs to reach disk at all.

class btclib_node.chainstate.block_index.BlockInfo(header: BlockHeader, index: int, status: BlockStatus = BlockStatus.valid_header, downloaded: bool = False)

Bases: object

One header this index has indexed: its height, status and download state.

header is the parsed header; index is its height; status and downloaded are this index’s own bookkeeping about it. chainwork is deliberately not a field here – the comment above argues why.

classmethod deserialize(data: bytes, *, check_validity: bool = True) → BlockInfo

Parse a BlockInfo from the bytes serialize produced.

serialize() → bytes

Serialize this record to the bytes stored under blkinfo-<hash>.

class btclib_node.chainstate.block_index.BlockStatus(*values)

Bases: IntEnum

Where a block stands relative to the active chain.

valid_header is a header on its own, content not yet checked; in_active_chain is on the active chain now; valid is a block whose content passed validation but that a reorg has since removed from the active chain (_finalize_fork’s own to_remove loop is the only place that sets it). invalid is terminal, set on a block itself or on any block built on one already marked invalid.

btclib_node.chainstate.block_index.calculate_work(header: BlockHeader) → int

Return the work header’s own target represents.

btclib_node.chainstate.contextual module

What the chain before a header requires of that header.

Bitcoin Core’s ContextualCheckBlockHeader and the two answers it asks for: GetNextWorkRequired in pow.cpp for the target, and CBlockIndex::GetMedianTimePast in chain.h for the timestamp. BlockHeader.assert_valid_pow answers the other half of the proof-of-work question – whether the hash meets the target the header itself claims – and needs no chain to do it, which is why one header carrying a target no chain hands out passes it.

The chain is reached through a callable that steps back one header, rather than through the index: a batch off the wire is checked before any of it is indexed, so its own members are what the header after them is checked against.

btclib_node.chainstate.contextual.assert_valid_in_context(chain: Chain, header: BlockHeader, parent: BlockHeader, parent_height: int, parent_of: ParentOf, now: datetime) → None

Assert what the chain before a header requires of it.

Six parameters and one call site: each is a distinct, independent piece of what Core’s own ContextualCheckBlockHeader reads too – the chain’s own rules, the header, its parent, that parent’s height, a way to walk further back for the two checks that need more than one ancestor, and the time to check the header’s own against. No subset of them travels together anywhere else in this module, so grouping any of them into a struct of their own would be a wrapper built for this one call rather than a shape the data already has.

Core’s ContextualCheckBlockHeader, in its order, less the version floors a deployment decides: bad-diffbits, then time-too-old, then time-too-new. parent sits at parent_height, so the header being checked is the block after it.

The timewarp rule Core adds under enforce_BIP94 is not here: it holds on testnet4 and on a regtest run with -test=bip94, and this node offers neither network.

btclib_node.chainstate.contextual.block_time(header: BlockHeader) → int

Return the second the header’s four timestamp bytes hold.

Core’s CBlockHeader::GetBlockTime. BlockHeader.serialize writes int(time.timestamp()), so that is the value the rules here compare: a header is weighed as it goes on the wire and not as it was built.

btclib_node.chainstate.contextual.header_at_height(header: BlockHeader, height: int, target_height: int, parent_of: Callable[[BlockHeader], BlockHeader]) → BlockHeader

Walk back from header, at height, to its ancestor at target_height.

Core’s CBlockIndex::GetAncestor over a skip list; this walks parent_of one header at a time, which is the same cost median_time_past below already pays to reach its own eleventh ancestor – unbounded here rather than capped at ten, since a caller asking for a BIP68 time-locked input’s own coin height can name any past height, not only one within the last eleven blocks. main.py’s own callers are where that cost is paid, once per input actually carrying a time-based relative lock rather than once per block regardless.

btclib_node.chainstate.contextual.median_time_past(header: BlockHeader, height: int, parent_of: Callable[[BlockHeader], BlockHeader]) → int

Return the median timestamp of a header and its ten ancestors.

Core’s CBlockIndex::GetMedianTimePast, whose window is however many of the eleven exist: nearer the genesis than that it is the whole chain, and the middle of an even number of times is the later of the two middle ones.

btclib_node.chainstate.contextual.next_bits_required(chain: Chain, parent: BlockHeader, parent_height: int, time: int, parent_of: ParentOf) → bytes

Return the compact target a header on this parent has to carry.

Core’s GetNextWorkRequired, asked of the parent at parent_height about a header timestamped time. The target moves once every DIFFICULTY_ADJUSTMENT_INTERVAL blocks and is the parent’s the rest of the time, so bits is a value the chain fixes rather than one a miner chooses.

A chain that does not retarget is answered first and with the parent’s target, where Core reaches the same answer further down: every branch it takes on such a chain answers with the limit, and the limit is what every block there carries, Chain.pow_limit_bits being the genesis’ own target and the genesis the block the rest descend from. Asking it here is what keeps the min-difficulty walk below off a chain on which each of its steps is one more block back to the genesis.

btclib_node.chainstate.filter_index module

The BIP158 filter of every connected block, and its filter header.

What a node holds that btclib.block.block_filter does not: the arithmetic over one block is btclib’s, and this is the index over the chain – one filter per block, and the header chaining it onto its parent’s.

A filter is keyed by block hash and so is its header, both being functions of the block and of its ancestry: a header is filter_header(this filter’s hash, the parent’s header) and the parent is fixed by previous_block_hash, not by which chain is active. So a block stepped over in a reorg keeps the filter and the header it had, and coming back costs nothing – which is why there is no counterpart here to UtxoIndex.apply_rev_block.

Writes are held until finalize, the way UtxoIndex holds them: a block connects in the same write batch as the chainstate it advances, and until that batch is written the parent of the block being indexed is not in the database yet.

BIP157 asks for exactly this: “Nodes SHOULD NOT generate filters dynamically on request, as malicious peers may be able to perform DoS attacks by requesting small filters derived from large blocks.”

class btclib_node.chainstate.filter_index.FilterIndex(parent_db: KeyValueStore, chain: Chain, logger: Logger)

Bases: object

The BIP157/BIP158 filter and header for every connected block.

The module docstring above is where the keying, the write-batch discipline and the lack of a reorg-time undo are all argued.

pending now survives more than one trial of main.update_chain, the same way UtxoIndex’s own staging does, so rollback cannot stay a blanket wipe either: add_block only ever adds a hash that was not already a key (its own guard, checking get_filter first), so _trial_log needs to remember only which hashes a trial added, not what they replace – UtxoIndex’s own _undo_log carries a prior value for exactly the mutations here that never happen.

add_block(block: Block, prevout_scripts: list[bytes]) → None

Index the filter of a block whose parent is already indexed.

add_connected_block(block: Block, rev_block: RevBlock) → None

Index a block from the reverse patch its connection produced.

RevBlock.to_add is the output every input of the block spent, which is what a filter needs and a block does not carry – prevout_scripts_from_utxos wants a bare TxOut per outpoint, one step short of what RevBlock.to_add now carries alongside it (Coin’s own docstring, block_db/__init__.py), so it is unwrapped here rather than asking that function to know about a type outside its own package.

catch_up(active_chain: list[bytes], block_db: BlockDB) → int

Index every block of the active chain that has no filter yet.

A datadir synced before this index existed has the blocks and the reverse patches and none of the filters, and a node that answers getcfilters for some of its chain is worse than one that does not answer at all – BIP157’s service bit is a promise about the whole of it. Walked from the bottom so that each block’s parent is indexed before it is.

Returns how many it built, which is zero on every start but the first after the index appears.

finalize(wb: KeyValueStore | None = None) → None

Write what is held, into wb if there is one and atomically.

The header before the filter, and the pair in one write. Both skip guards ask get_filter, so a filter written without its header is the state nothing repairs: the block is skipped for ever and its child cannot be indexed at all, which leaves the datadir unopenable. The other half of the pair is harmless – a header with no filter is simply rebuilt.

get_filter(block_hash: bytes) → bytes | None

Return the serialized filter of a block, or None.

get_filter_hash(block_hash: bytes) → bytes | None

Return the filter hash of a block, in display order, or None.

get_header(block_hash: bytes) → bytes | None

Return the filter header of a block, or None.

rollback(mark: int = 0) → None

Discard whatever pending gained since mark.

mark defaults to the very start, which is every direct test of this method: a fresh index, nothing pending before its own trial. UtxoIndex.rollback’s own docstring is where a caller with something to protect gets a real one from.

trial_mark() → int

Return a point in the trial log a failed trial can be rolled back to.

UtxoIndex.trial_mark’s own docstring is where the reason this exists at all – pending surviving more than one trial – is argued; main.update_chain is the one caller.

btclib_node.chainstate.muhash module

A rolling, order-independent commitment to the whole UTXO set.

Core’s MuHash3072 (src/crypto/muhash.h/.cpp, at bitcoin/bitcoin@ca7162cde5) represents a multiset as a fraction of two 3072-bit numbers modulo the largest 3072-bit safe prime, 2**3072 - 1103717: inserting an element multiplies it into the numerator, removing one multiplies it into the denominator, and the two operations are exact inverses of each other regardless of order or of what else has been inserted or removed meanwhile – the property UtxoIndex below relies on for rollback. MuHash3072 here is that same construction, Insert/Remove/Finalize renamed insert/ remove/digest – lowercase, matched against Core’s own muhash_tests (src/test/crypto_tests.cpp) and against RFC 8439’s own ChaCha20 vectors before anything is built on it (tests/unit/chainstate /muhash_test.py).

The arithmetic is native Python int: pow, % and pow(x, -1, m) for the modular inverse Finalize needs, in place of Core’s own limb-by-limb Num3072 – a fixed-width C++ integer split into 32- or 64-bit limbs for a CPU that has no 3072-bit register, which Python’s own arbitrary-precision int already is without that machinery. This is the Python-native licence CLAUDE.md’s own Following Bitcoin Core gives: the arithmetic is native rather than reimplementing Core’s own carry-and-reduce trick, and what is committed to – the modulus, the per-element hash, the byte order – is unchanged. One divergence this buys for free rather than by design: Num3072’s own “overflow” state (a value held between the modulus and 2**3072, only reduced lazily) has no counterpart here, because % _MODULUS after every multiply keeps _numerator/_denominator always fully reduced – cheaper in Python than replicating the lazy reduction, and crypto_tests.cpp’s own overflow vector is still matched (muhash_test.py), since a value Core would carry unreduced and only fold in at Finalize is folded in here immediately instead, with the same result either way.

## The per-element hash

_num3072, matching MuHash3072::ToNum3072: SHA256 of the element’s own bytes (HashWriter::GetSHA256, a single SHA256, not Bitcoin’s usual double one) keys a ChaCha20 stream cipher seeded at nonce zero and block counter zero (ChaCha20Aligned’s own default, SetKey), whose first 384 bytes of keystream are read as one little-endian 3072-bit integer – Num3072::ToBytes/its constructor pack each 64-bit limb little-endian and the limbs least-significant first, which is exactly int.from_bytes(data, “little”) over the whole 384 bytes at once. _chacha20_block below is RFC 8439’s block function – the same QUARTERROUND rotation amounts (16, 12, 8, 7) and the same column-then-diagonal ordering chacha20.cpp’s own unrolled REPEAT10 carries – fed the block counter in word 12 and an all-zero 96-bit nonce in words 13-15, matching ChaCha20Aligned::Seek’s layout at the (0, 0) nonce and 0 counter ToNum3072 never overrides. Six blocks (_KEYSTREAM_BLOCKS) cover the 384 bytes Num3072::BYTE_SIZE names; the 32-bit block counter never overflows into the nonce word Core’s own ++j12; if (!j12) ++j13; carries into, six being nowhere near 2**32.

This is the dominant cost of connecting a block, measured directly against a block the shape btclib-org/btclib-node#586 already measured (964,000’s own 7,778 spends and 8,100 creations): the SHA256-then-six- ChaCha20-blocks this section builds, purely in Python, costs low-hundreds of microseconds per coin, on the same order as or larger than add_block’s own remaining work for a block that dense. Left as pure Python for now regardless – CLAUDE.md’s own Python-native licence – with the number itself in the pull request rather than here, where it would age; a compiled ChaCha20 is a decision for whoever reads that number next, not one this module makes for them.

## What is inserted, and what is not

CoinStats bundles the accumulator with the three running counters Core’s own -coinstatsindex (src/index/coinstatsindex.cpp, same sha) keeps beside it – transaction_output_count, total_amount, bogo_size – because gettxoutsetinfo reports all four from the same incrementally-maintained state rather than a fresh scan (rpc/callbacks.py’s own get_tx_out_set_info is where the two are told apart). tx_out_ser matches TxOutSer (src/kernel/coinstats.cpp): the outpoint (36 bytes, already this tree’s own wire serialization), the packed (height << 1) | coinbase as a fixed 4-byte little-endian `uint32` – unlike Coin.serialize in block_db/__init__.py, which packs the same field as a var_int for storage density, Core’s TxOutSer uses ss << (uint32_t)packed, fixed width, and the two encodings are not interchangeable – and the output itself, TxOut.serialize, byte for byte CTxOut’s own. bogo_size matches GetBogoSize (kernel/coinstats.cpp): a fixed 50 bytes plus the script’s own length, not the script’s actual serialized size (Core’s own comment calls it a “database-independent metric” for a reason – it is not meant to reproduce a wire size).

is_unspendable matches CScript::IsUnspendable (script/script.h:564-567, same sha): a leading OP_RETURN (0x6a) or a script over MAX_SCRIPT_SIZE (10,000 bytes). CCoinsViewCache ::AddCoin (coins.cpp:82) returns without ever adding such an output to Core’s own UTXO set in the first place, so ApplyCoinHash never sees one either. CoinStats.insert/remove gate on it independently of what UtxoIndex’s own store keeps – which is what already made the digest and the three counters match Core’s on a block carrying an OP_RETURN output before UtxoIndex’s own add_block gated on it too (btclib-org/btclib-node#667): even while the two disagreed on membership, neither ever offered an unspendable output to ApplyCoinHash/CoinStats.insert, so the two accumulators agreed regardless. UtxoIndex.add_block now also skips storing such an output under a utxo- key at all, matching AddCoin’s own refusal directly rather than only agreeing with it through CoinStats’s own independent gate (utxo_index.py’s own _stage_creation is where that gate, and apply_rev_block’s own consequence – an output never stored is never restored – are argued); gettxoutsetinfo’s txouts and bogosize (rpc/callbacks.py) were always CoinStats’s own count, never a scan of the utxo- namespace, so this changes nothing either answers.

## The two blocks history exempts

IsBIP30Unspendable (validation.cpp:6224-6228, same sha) names two mainnet blocks, 91722 and 91812, each mined before BIP34 gave a coinbase’s own outpoint a height it could never collide with, whose coinbase transaction was later duplicated verbatim by a different block (91842, 91880 – IsBIP30Repeat, the pair chains.py’s own Chain.bip30_exceptions names, which is what _check_bip30 on the second occurrence is waived against, not this one). CoinStatsIndex::CustomAppend skips a duplicated coinbase’s own outputs entirely – never inserted, so never later removed either – on whichever of the two connects first carrying that flag: the first occurrence is never hashed, so the second occurrence’s own ordinary insertion is the only one the accumulator ever carries, and the outpoint’s later spend correctly cancels exactly that one. Reproduced here as _BIP30_UNSPENDABLE_ORIGINALS, checked once per connecting block’s own coinbase in UtxoIndex.add_block – chains.py is other work’s own region for this branch, so the pair is local rather than a new Chain attribute. Both blocks are ninety-odd thousand mainnet blocks deep and neither height nor hash is reachable on any chain this tree’s own test suite runs (Chain.bip30_exceptions is empty on every chain but mainnet), so this exclusion is matched against Core’s source rather than against a live run of it.

class btclib_node.chainstate.muhash.CoinStats(muhash: MuHash3072 = <factory>, transaction_output_count: int = 0, total_amount: int = 0, bogo_size: int = 0)

Bases: object

The MuHash accumulator plus Core’s own three running counters.

transaction_output_count, total_amount and bogo_size are CoinStatsIndex’s own m_transaction_output_count, m_total_amount and m_bogo_size (index/coinstatsindex.cpp, same sha) – maintained the same way the accumulator is, incrementally, rather than by scanning the set rpc/callbacks.py’s get_tx_out_set_info answers from.

classmethod deserialize(data: bytes) → CoinStats

Parse the bytes serialize produced.

digest() → bytes

Return the 32-byte commitment – self.muhash.digest().

insert(out_point_bytes: bytes, coin: Coin) → bool

Insert coin; False and a no-op if it is unspendable.

The module docstring’s own “What is inserted, and what is not” argues the gate; UtxoIndex._hash_insert is the one caller, and remove below is its exact undo regardless of this return value – a coin this skips is skipped identically on the way back out.

remove(out_point_bytes: bytes, coin: Coin) → bool

Remove coin; False and a no-op if it is unspendable.

serialize() → bytes

Return this tree’s own shape: the muhash, then three counters.

Each counter is 8 bytes, signed, big-endian. KeyValueStore’s own meta column family is where UtxoIndex.finalize writes this, in the same write_batch as the coins it commits to (db.py’s docstring argues why).

class btclib_node.chainstate.muhash.MuHash3072(numerator: int = 1, denominator: int = 1)

Bases: object

A running numerator/denominator over _MODULUS – Core’s MuHash3072.

The module docstring above is where the construction, the per-element hash and the “no overflow state” divergence are all argued.

classmethod deserialize(data: bytes) → MuHash3072

Parse the 768 bytes serialize produced.

digest() → bytes

Return the 32-byte commitment – MuHash3072::Finalize.

Core’s own Finalize divides the numerator by the denominator in place and resets the denominator to one – a normalization that leaves the represented value unchanged (its own comment says so) but is otherwise pure bookkeeping, so this returns the digest without mutating self: every caller here (CoinStats .digest, muhash_test.py) reads a value that keeps accumulating afterwards, unlike Core’s own single-use MuHash3072 acc locals in crypto_tests.cpp.

insert(data: bytes) → None

Multiply data into the numerator – MuHash3072::Insert.

remove(data: bytes) → None

Multiply data into the denominator – MuHash3072::Remove.

The exact inverse of insert on the same bytes, in either order and regardless of anything else inserted or removed meanwhile: insert/remove only ever multiply the numerator and the denominator independently, and a factor common to both cancels out at digest’s own division whatever else multiplied either one in between. UtxoIndex.rollback below relies on exactly this to undo a staged insert with a remove and vice versa, without recording what the accumulator’s own state was before either.

serialize() → bytes

768 bytes: the numerator, then the denominator, each 384 bytes LE.

This tree’s own on-disk shape, not Core’s – MuHash3072 ::SERIALIZE_METHODS serializes the same two numbers in the same order, matched here only because the shape is the obvious one, not because anything reads this file’s bytes with Core’s own code.

classmethod singleton(data: bytes) → MuHash3072

Build a set holding exactly data – the MuHash3072(span) ctor.

btclib_node.chainstate.muhash.chacha20_keystream(key: bytes, blocks: int, *, nonce_words: tuple[int, int, int] = (0, 0, 0), counter: int = 0) → bytes

blocks blocks (blocks * 64 bytes) of ChaCha20 keystream.

nonce_words and counter default to zero – ChaCha20Aligned’s own default, matching MuHash3072::ToNum3072’s usage (ChaCha20Aligned{key}.Keystream(…), no Seek call), which is the only way _num3072 below ever calls this. The two arguments exist so muhash_test.py can drive this same function against crypto_tests.cpp’s own RFC 7539/8439 vectors, seeked to a real nonce and counter this tree never needs otherwise – including the 32-bit counter overflow that carries into nonce_words[0], matching chacha20.cpp’s own ++j12; if (!j12) ++j13;.

btclib_node.chainstate.muhash.is_bip30_unspendable(height: int, block_hash: bytes) → bool

Report whether block_hash/height is one CoinStatsIndex skips.

The module docstring’s own “The two blocks history exempts” argues why, and chains.py’s own Chain.bip30_exceptions is the different pair this is not.

btclib_node.chainstate.muhash.is_unspendable(script: bytes) → bool

Report whether script can never be spent – IsUnspendable.

btclib_node.chainstate.muhash.tx_out_ser(out_point_bytes: bytes, coin: Coin) → bytes

Return the bytes ApplyCoinHash/RemoveCoinHash insert – TxOutSer.

out_point_bytes is the 36-byte wire outpoint a caller here already has (OutPoint.serialize, byte for byte COutPoint’s own); the packed (height << 1) | coinbase is a fixed 4-byte little-endian uint32, not the var_int Coin.serialize (block_db/__init__.py) packs the same field as for on-disk density – the module docstring above is where that divergence from this tree’s own storage format is argued. coin.tx_out.serialize closes it out, byte for byte CTxOut’s own.

btclib_node.chainstate.utxo_index module

UtxoIndex, the set of transaction outputs a spend may still reference.

add_block applies one block’s own spends and creations, returning the prevouts each transaction consumed – what interpreter.check_transactions validates against – and the block_db.RevBlock a reorg away from this block would need to undo it.

class btclib_node.chainstate.utxo_index.UtxoIndex(parent_db: KeyValueStore, logger: Logger)

Bases: object

The set of spendable outputs, staged in memory until finalize.

removed_utxos and updated_utxo_set hold what a batch of add_block/apply_rev_block calls has changed since the last finalize; the module docstring above is where add_block’s own return value is argued. The staging now survives more than one block – should_flush is what tells a caller it is time to stop piling more of it on and write, and db.py’s docstring is where what a crash before that costs is decided.

That survival is what makes rollback unable to stay a blanket wipe: a trial main.update_chain rolls back may run against staging several earlier, already-succeeded blocks left behind, unflushed, and wiping the two dicts to empty would discard those too – state a failed trial never touched and has no claim over. _undo_log is what tells the two apart: every mutation add_block and apply_rev_block make is recorded there as it happens, and rollback(mark) undoes only what was recorded since mark (trial_mark’s own reading, taken before the trial that might fail began), in reverse, leaving anything recorded before it standing.

coin_stats is the running commitment to the same set – chainstate/muhash.py’s own CoinStats, restored from parent_db’s meta column family here and staged the same way the two dicts above are: every _hash_insert/_hash_remove this class makes is logged into the same _undo_log, CoinStats.insert and .remove being each other’s exact undo regardless of order (muhash.py’s own docstring argues why), so rollback needs no prior accumulator state recorded, only that the opposite call replays.

add_block(block: Block, height: int, *, check_bip30: bool = True) → tuple[list[tuple[list[Coin], Tx]], RevBlock]

Apply block’s own spends and creations, staged rather than written.

height is this block’s own height, on whichever branch it is being tried – what every output it creates is stamped with, coin and coinbase alike, and never the height a later reorg disconnects or reconnects it at: apply_rev_block below restores a Coin exactly as this call staged it for removal, height and coinbase bit included, rather than recomputing either.

check_bip30 refuses a block that “overwrites” an output still unspent from anywhere earlier on the chain – Core’s own bad-txns-BIP30 (ConnectBlock, src/validation.cpp:2401-2431, at bitcoin/bitcoin@204256c73f), CVE-2012-1909’s shape: without it, a coinbase sharing an already-mined, still-unspent txid overwrites that output in place, and a reorg away from the second block deletes an output the first block’s own branch still carries. Checked over every transaction the block carries, coinbase included, matching Core’s own loop – and before either of the two loops below stages a single write, since Core’s own check runs against the view exactly as it stood before this block, coinbase and ordinary spends alike. False only for the two 2010 blocks Chain.bip30_exceptions names, which predate BIP34 (btclib-org/btclib-node#571) and so predate the property that makes a new violation of this kind unreachable once BIP34 binds: a block’s own coinbase commits to its own real height, which two different heights can never share, so the outpoint a block’s own coinbase creates cannot already belong to an earlier block’s coinbase – and UtxoIndex.add_block un-stages an entire block atomically on any raise, this one included, so a refused duplicate never reaches the two loops below that would otherwise stage a write over it.

Both loops below call _unmark_removed on an outpoint’s own bytes before every _put of it, the same order apply_rev_block’s own to_add loop uses to restore one: a key this call creates can coincide with one removed_utxos still carries only when the two share a txid, check_bip30 above being what refuses that for every case but the two historical exceptions – so the unmark is a no-op everywhere else, and is what keeps removed_utxos and updated_utxo_set disjoint rather than leaving a recreated outpoint staged in both at once (_bip30_violation’s own docstring is where that invariant is used, and btclib-org/btclib-node#586 is where staging it in both broke a later apply_rev_block on a coin that was legitimately unspent).

Returns each non-coinbase transaction paired with the prevouts its own inputs consumed – what interpreter.check_transactions validates against – and the RevBlock that undoes this call.

Every output this stages and every prevout it spends also moves coin_stats, the running commitment to the set – inserted for a creation, removed for a spend, Coin.parse’s own two raises above unaffected since they run before either ever touches it. The coinbase creation loop’s own _hash_insert is skipped for the two mainnet blocks is_bip30_unspendable names, matching CoinStatsIndex::CustomAppend – muhash.py’s own “The two blocks history exempts” argues why, and why this is a different pair from check_bip30’s own exception above.

Neither creation loop stages a provably unspendable output at all – _stage_creation’s own docstring is where that gate, and why it is what keeps added (and so RevBlock.to_remove) from ever naming an outpoint this call never wrote, is argued (btclib-org/btclib-node#667). A block’s own spend loop below needs no matching gate: no valid witness satisfies a provably unspendable output’s own script, so no block this method is ever asked to connect legitimately spends one, and one that tried would find it absent – “prevout not found”, the same answer Core’s own ConnectBlock reaches through Consensus::CheckTxInputs/HaveInputs for an output its own AddCoin never added either.

apply_rev_block(rev_block: RevBlock) → None

Undo add_block for the block rev_block was returned for.

Removes every outpoint it created and restores every prevout it spent, staged the same way add_block stages its own changes.

to_add runs before to_remove, not the reverse order add_block itself builds the two lists in, because an ordinary chained transaction – one spending an output another transaction earlier in the same block created – puts that output’s outpoint in both: to_remove from being created, to_add from being spent before this block ever finalized it to disk. Popping it in to_remove first would look it up while it is in neither updated_utxo_set nor the database – its net effect on the persisted set is nothing, both before this block and after it – and raise “output not found” on a block that did nothing wrong. Restoring it in to_add first stages it back into updated_utxo_set, where to_remove’s own _pop then finds and removes it, netting to the same nothing add_block itself computed. Every other entry is unaffected by the order: to_add’s outpoints predate this block and never collide with to_remove’s own, which this block alone created, a valid block spending a given outpoint at most once. Core’s DisconnectBlock (src/validation.cpp, at bitcoin/bitcoin@05e49b342f) reaches the same result walking one transaction at a time in reverse block order – spend its own outputs, then restore its own inputs – rather than in the two flat passes here (btclib-org/btclib-node#634).

A restored prevout is unmarked from removed_utxos before it is put back, not merely put back: add_block staged that spend with _mark_removed whenever the prevout was already durable (found in self.db rather than in updated_utxo_set), and leaving that flag set here would put the same outpoint bytes in both removed_utxos and updated_utxo_set at once – still “removed” as far as a later add_block call’s own guard is concerned, even though this call just made it spendable again. Staging now survives across trials (this outpoint can sit restored for up to _FLUSH_BOUND entries’ worth of blocks before finalize clears both dicts), so a stale flag here is no longer erased by the next trial boundary the way the old, per-trial finalize used to erase it – it stays wrong until a legitimate later spend of the same output hits add_block’s “prevout already spent in this batch” guard and gets rejected as a double spend, invalidating that block and, through update_header_index -> BlockIndex.invalidate, everything built on it. Independently of that, the same stale flag hides a genuine BIP30 duplicate too: _bip30_violation reads removed_utxos first and answers “no violation” on a hit, so a block recreating the restored outpoint – still unspent once this call has put it back – would connect instead of being refused bad-txns-BIP30 (btclib-org/btclib-node#586).

coin_stats moves the opposite way add_block moved it for the same coin: to_add restores a prevout that block spent, so it is inserted back into the commitment; to_remove drops an output that block created, so it is removed from it – both skipped for a coin that is both coin.is_coinbase and named by is_bip30_unspendable, matching add_block’s own gate, which withholds only the coinbase creation loop (skip_coinbase_hash, add_block’s own docstring) and never an ordinary transaction sharing the same block. is_coinbase alone is what is_bip30_unspendable cannot answer for: it checks only the coin’s own height and the block’s hash, so without this condition every non-coinbase output of the exempt block itself – stamped with the same height by add_block – would be wrongly withheld here on undo despite having been correctly hashed in on the way in, leaving coin_stats permanently off after a reorg through that block. Matches CoinStatsIndex::CustomAppend’s own is_coinbase && IsBIP30Unspendable(…) (src/index/coinstatsindex.cpp:129, at bitcoin/bitcoin@ca7162cde5), which gates on exactly the same conjunction rather than the hash/height pair alone. to_remove parses the stored Coin for this alone where the rest of this loop only ever needed to know the record existed. A parse failure here raises ChainstateInconsistencyError, unlike the identical fault reached through _stored_prevout (add_block’s own prevout resolution, get_coin), which answers None instead (btclib-org/btclib-node#650, _stored_prevout’s own docstring). The two share only the attribution – that the fault is this node’s own corrupted record, never a caller’s content (btclib-org/btclib-node#620, btclib-org/btclib-node#631, btclib-org/btclib-node#636) – not the outcome: undoing a block this node already connected has no legitimate “not found” reading the way an ordinary prevout lookup does, this loop’s own “output not found” raise immediately above already treating absence itself as this node’s own bookkeeping fault rather than a candidate’s.

finalize(wb: KeyValueStore | None = None) → None

Write every staged change into wb, or into self.db if none.

Everything staged is durable after this, so nothing recorded before it can ever be rolled back to: _undo_log is cleared along with the two dicts it was tracking. coin_stats is written alongside them, into the same store’s meta column family (db.py’s own put_meta) – inside wb’s own batch when a caller passes one, which is what keeps the commitment and the coins it commits to landing together or not at all (db.py’s docstring argues why).

get_coin(prevout_bytes: bytes) → Coin | None

Return the Coin a serialized outpoint still resolves to, or None.

Checks updated_utxo_set and removed_utxos first, the same order add_block and apply_rev_block already read staged state in, before falling to the store: a coin several blocks’ own worth of staging have created or already taken is real regardless of whether finalize has written it out yet, and a caller reading self.db directly – main.verify_mempool_acceptance used to – would miss exactly what staying staged across more than one block (btclib-org/btclib-node#586) makes ordinary.

rollback(mark: int = 0) → None

Undo every mutation recorded since mark, in reverse.

mark defaults to the very start – a caller with nothing staged before its own trial began, which is every direct test of this method – and trial_mark’s own docstring is where a caller with something to protect gets a real one from.

should_flush() → bool

Whether the staged size has reached _FLUSH_BOUND.

main._finalize_fork is the one caller, and it is not asking this alone: Chainstate.flush writes BlockIndex and FilterIndex in the same batch this triggers, which is what keeps the three in step (db.py’s own docstring argues why).

trial_mark() → int

Return a point in the undo log a failed trial can be rolled back to.

Taken before add_block/apply_rev_block are ever called for that trial; rollback undoes back to exactly this point, main.update_chain’s own docstring above being the one caller.

Module contents

Chainstate: the block index, the UTXO set and the compact filter index.

All three share the one db.KeyValueStore Chainstate opens, told apart by each’s own key prefix – db.py’s own docstring is where that shared store’s key order is argued. block_index.BlockIndex tracks headers and which chain is active, utxo_index.UtxoIndex the spendable outputs on it, and filter_index.FilterIndex the BIP157/BIP158 filters served over p2p; contextual.py is the height- and time-dependent validation the first of those calls before extending the active chain.

flush is what writes all three indexes’ own staged changes in one batch, and close calls it before closing the store – db.py’s docstring is where the crash this is the other half of is argued.

class btclib_node.chainstate.Chainstate(data_dir: Path, chain: Chain, logger: Logger)

Bases: object

The block index, the UTXO set and the compact filter index, together.

The module docstring above is where the three, and the one KeyValueStore they share, are argued.

close() → None

Flush every staged index, then close the shared store.

A clean close loses nothing staged – the crash this store has to survive is one that never reaches this method at all, and db.py’s docstring is where what that crash costs is decided. Safe to call twice: flush needs the connection open, so a second call skips it and reaches only KeyValueStore.close’s own no-op on an already-closed store.

flush() → None

Write every index’s own staged changes, in one atomic batch.

main._finalize_fork stages a connected or disconnected block’s own status (BlockIndex.stage_status) and its filter (FilterIndex.add_connected_block) the same way UtxoIndex already staged its spends and creations, across more than one block; writing the three together here – one write_batch, one commit – is what keeps a status or a filter from ever landing on disk ahead of the UTXO set it was validated against. db.py’s own docstring argues why that has to hold.