backend/transactions: add test for tx serialization

[Beerosagos]

Recent btcd updates changed the way hashes are serialized. We proposed a fix (see https://github.com/btcsuite/btcd/pull/2025) to restore backward compatibility.

This commit adds a specific test case to ensure that the newer and the legacy serialization methods work.

[Beerosagos]

Moved the transactionsdb_test.go and made a few changes, PTAL :pray:

[Beerosagos]

rebased