signature – manually signing bitcoin transaction


I’m trying to create a BTC transaction with one input and two outputs, one for change and one OP_RETURN to anchor data. And I’m doing all of that on my regtest backend. The raw unsigned transaction is

01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c0000000023210340165231215a98e7a32abce9d410ecd09ac505938b25f9451defa051d591ebf8acffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a408973d6b447bdda5312b1ef1b5509668672296301dd328d4e55317def98f165d200000000

I can then use the bitcoin rpc to sign it and it works, so I’m pretty certain that this part is correct. When I however try to sign it manually I get the error

500: {"result":null,"error":{"code":-26,"message":"16: mandatory-script-verify-flag-failed (Non-canonical DER signature)"},"id":2}

I’ve followed the steps outlined in How to redeem a basic Tx? as well as Signing a raw transaction with Python ECDSA (or OpenSSL) as well as attempted to use pybitcointools to sign the input without success.

The address and private key I use are

addr: "mmEXEzUGcMmmiLsfxxM8gB8TQSTkuR1drf"
pk:  "cTyF9pebH3kwwzUt5gzaxSDQ1DbqYfx4P1i4d1TyjtSDEeUFgYsk"

and stragely, even though the PK is the one given to me by ./bitcoin-cli -regtest dumpprivkey <addr> does not seem to match the address and I’m not sure why or what it is I need to do to it (it’s in WIF format and compressed, i.e. 0x01 appended to the end

The original TX includes the scriptPubKey of the utxo. I then double sha256 the whole raw tx (shown above) and replace the scriptPubKey part with the sigScript | pubkey using the following code:

private_key, compressed_pk = wif_to_private_key(private_key_wif)
tx = bytearray.fromhex(raw_unsigned_transaction) + int(1).to_bytes(4, 'little')

double_sha256_tx = hashlib.sha256(hashlib.sha256(tx).digest()).digest()
signing_key = ecdsa.SigningKey.from_string(bytearray.fromhex(private_key), curve=ecdsa.SECP256k1)
public_key = bytearray.fromhex(privtopub(private_key_wif))
signature = signing_key.sign_digest(double_sha256_tx, sigencode=ecdsa.util.sigencode_der) + int(1).to_bytes(1, 'little')
scriptSig = to_varstr(signature) + to_varstr(public_key)

to_varstr simply prints (len)|(data) and privtopub is the function from pybitcointools as I couldn’t figure out how to go from the compressed private key to the address manually. It’s worth noting though that the public_key here that privtoaddr from pybitcointools does not return the correct address either which might be the root cause of the issue?

I’d be grateful for any pointers or things to try

cheers
Chris

PS:
another interesting aspect is that the signed transaction I generate is a lot longer than the one generated by the signtransaction rpc call:

rpc:    01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c0000000023210340165231215a98e7a32abce9d410ecd09ac505938b25f9451defa051d591ebf8acffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a40ac4f0818b683eeeaa1fbf2f508af2fc22cc814e69025152c70d7c414ebbfc30a00000000
manual: 01000000018be9d0e99e74d69d915e105db1328707f713d42a894909b18a78fe68e1d8290c000000006b483045022100d6e538aa819f3162d5c6a0e4d9ee0323395df89e943c769d96ae939baec5c6920220083a311cb35df3c98d7a6bd0bc80d98a71fbdf8e325ba908ea4b721a4eac8bb301210343395a6e84c7f2b1d50c11f96783664a6f04d66b51befb3befcc57334e2a9abcffffffff0218ee052a010000001976a9143eb52fb0be4be87edc74848b371547f663e26c7e88ac0000000000000000226a40ac4f0818b683eeeaa1fbf2f508af2fc22cc814e69025152c70d7c414ebbfc30a00000000