btcd
We’ve just released btcd version 0.12.0 beta! It is a highly recommended updated since it contains code to ensure new upcoming network rules are properly enforced (CLTV), new protocol enhancements, various optimizations, and several other enhancements.
Overview
In addition to announcing the new release, the purpose of this blog is to discuss a few points of the release in more detail. For those of you who are simply looking for the release notes and updated Windows binaries, they are located here.
The most important changes this post will cover are:
CHECKLOCKTIMEVERIFY(CLTV) support (BIP0065)- Enforce
LowSsignatures for transaction relay - New flag for controlling the minimum relay fee
- Bloom filter service bit support (BIP0111)
- New signature validation cache
- Developer-centric changes:
- Refactored peer package
- New options when querying address index
- Under development
CHECKLOCKTIMEVERIFY Support (BIP0065)
This release implements BIP0065 which introduces a new opcode named CHECKLOCKTIMEVERIFY that provides the scripting system a method to guarantee a transaction is unspendable until a specific date, or block height, in the future. While transactions already contained a lock time field, there was no way to enforce it via the scripting system prior to this new opcode.
The deployment of this new opcode is done via a soft fork using the same voting mechanism employed by BIP0034 and BIP0066 and entails adding a new block version 4 which treats the NOP2 opcode as CHECKLOCKTIMEVERIFY and enforces the new lock time restrictions.
In addition, all block templates returned by the getblocktemplate RPC will now be block version 4 compliant. This means solo miners using the getblocktemplate RPC will need to ensure their preferred software is updated to support the new block version and rules. Miners using a pool should follow the instructions provided by their pool operator.
Enforce LowS Signatures For Relay
ECDSA signatures are now required to be encoded with the canonical ‘low-s’ encoding in order to be relayed or accepted into the memory pool. This is useful since it helps reduce the number of malleable transactions relayed across the network and included in blocks.
However, please note that this is only a standardness check as opposed to a consensus change. This is an important distinction because it means it is still possible for miners to include signatures which do not conform to this encoding in blocks and therefore software must not assume old style signatures are impossible.
An important consequence of this change is that old software which creates transaction signatures that do not conform to this rule will need to be updated to produce transactions which are not rejected. That said, there are a wide variety of conforming implementations already available including the btcsuite signing code which has already produced signatures conforming to this encoding since Jul 31, 2014.
New Flag for Controlling the Minimum Relay Fee
As many users have probably noticed, there has recently been an increasing number of low-fee transactions flooding the network which causes the memory pool size to grow tremendously.
As a temporary measure to help mitigate this, a new parameter, --minrelaytxfee, has been added to allow the minimum transaction fee (in BTC/kB) transactions must have before they are allowed to be accepted into the memory and relayed to other nodes to be changed. The default value is 0.00001. Users with memory constrained systems, or who simply desire the memory pool to consume less space, will probably want to raise this to something higher such as 0.00005.
We are aware that this is not the best solution and plans are underway to provide a more robust long-term solution.
Bloom Filter Service Bit Support (BIP0111)
A new service flag which conforms to BIP0111 has been introduced into the peer-to-peer protocol which can be used to signal support for bloom filters. The is accompanied by the protocol version being increased to 70011.
Please note that peers running previous protocol versions that are newer than the version which bloom filters were introduced are assumed to support bloom filters since they were previously required.
To accompany this change, a new --nopeerbloomfilters parameter is now available which can be used to disable support for bloom filters and disable the new service bit accordingly.
A full discussion about why one might or might not want to disable bloom filters is beyond the scope of this post. Suffice it to say there are several issues with them such that many of us in the community would ultimately like to see them replaced. However, it is important to note that the vast majority of SPV implementations currently rely on full nodes having bloom filtering enabled, so we don’t recommend disabling them yet.
New Signature Validation Cache
This release adds a signature cache to the signature validation system which, as the name implies, caches successful signature validations in memory. This typically provides a significant speedup to block processing once the node is fully synced. This is the case because most transactions in new blocks have already been seen and validated by the memory pool and therefore the new blocks can avoid many relatively expensive signature checks by consulting the cache for the ones already known to be valid.
The cache is limited by a configurable number of entries (though future development will likely change this to a target memory size) which is set to 50000 entries by default. A new parameter, --sigcachemaxsize, has been added to allow the cache size to be configured.
Refactored Peer Package
NOTE: This section is aimed more at developers than users, so users may want to skip this section.
This release includes a complete refactoring of the peer code into a peer package which acts a common base for creating and concurrently managing bitcoin network peers. While it doesn’t directly have a huge impact on the current release, we are excited to announce this package because it paves the way for a wide variety of applications that work with the peer-to-peer network. For example, it could be used to easily create robust and more intelligent network seeders, SPV wallets, testing tools, relay nodes, proxies, etc.
As it specifically pertains to btcsuite, btcd has already been updated with this release to use the package. This is significant because it means the peer package will necessarily have to be kept up to date with the latest network and protocol changes that are required by the backbone of the network. Compare this model with having decoupled peer code (such as a 3rd-party library written in Python, node.js, etc) which is far more likely to constantly get out of sync with the latest core network development.
New Options When Querying Address Index
NOTE: This section is aimed more at developers than users, so users may want to skip this section.
The searchrawtransactions RPC which can be used to query the optional address index now includes three new parameters:
- An optional
vinextraparameter which causes the results to include information about the outputs referenced by a transaction inputs - An optional
reverseparameter to return the results in reverse order which can be useful in block explorer type of applications - An optional
filteraddrsparameter that filters out all transaction inputs and outputs that do not involve the provided addresses. This can significantly reduce the amount of data returned when the transactions involved have a lot of inputs and outputs and your application is only interested in those which involve a specific set of addresses as is typically the case
What’s Under Development?
The following is a quick overview of the major ongoing and future development plans:
- Major database restructuring and optimization
We have been keeping a close eye on the performance characteristics of btcd and noticed some areas, particularly in regards to the database, that aren’t scaling as well as we would like. Our goal is to eliminate these bottlenecks.
- Add support for concurrent syncing
Currently, all initial block syncing makes use of a single peer. This can lead to wild variance in sync time when the selected sync peer is stalled and/or slow. By making use of multiple peers, the initial sync time can be significantly reduced and other issues related to only having a single sync peer will be eliminated. The peer package introduced in this release was a key piece towards this effort.
- Improvements to headers-first syncing
While btcd has already supported headers-first syncing since version 0.5.0-alpha, it currently only uses that mode up until the final checkpoint. Our goal is to extend this capability beyond the final checkpoint.
- Refactor the memory pool code into a separate package
The current code which deals with the transaction memory pool is fairly tightly integrated into the main server code. We plan to refactor this out into a separate package which will greatly improve its testability, allow independent benchmarking and profiling, and open up some interesting opportunities for future development related to the memory pool.
- Refactor the mining code into a separate package
Much like the memory pool, the current code which deals with providing mining-related functionality, such as generating block templates and creating new blocks, is fairly tightly integrated into the main server and memory pool code. By separating this out into an independent package, many of the same benefits mentioned for the memory pool apply here as well.
- Several improvements to the memory pool
The current memory pool is not limited which can lead to it consuming massive amounts of memory. While on the surface it might seem simple to limit it to a specific number of entries, there are several factors that need to taken in account such as priority, fees, dependencies on other transactions, and general fairness to all involved parties.
As many readers following our releases probably have noted, the first three of these were already mentioned in the previous release. They have been repeated here since they are still ongoing and embody the major development efforts.