ISUP multipart encoding issue

[mkpaz]

I send HEP packets that contains a SIP message with ISUP attachment like this:

436f6e74656e742d446973706f736974696f6e3a207369676e616c3b68616e646c696e673d72657175697265640d0a0d0a << boundary
011048000a03020a0804108764125250f53904c0d03dc00a0884138764020090093d011bc00906841387640200900900 << ISUP binary
0d0a2d2d7369702d626f756e646172792d4d7a316753574e5756542d2d << boundary

I compared it with the actual SIP message and they're identical and can be decoded by Wireshark dissector.

But after inserting into PgSQL database it looks like this:

436f6e74656e742d446973706f736974696f6e3a207369676e616c3b68616e646c696e673d72657175697265640d0a0d0a << boundary
0110480a03020a0804106412525039043d0a08136402093d011b090613640209 << ISUP binary
0d0a2d2d7369702d626f756e646172792d4d7a316753574e5756542d2d << boundary

Some bytes just lost in the process:

011048000a03020a0804108764125250f53904c0d03dc00a0884138764020090093d011bc00906841387640200900900 << hep
0110480  a03020a080410  64125250  3904    3d  0a08  13  64020    93d011b 090 6  13  64020 9 << database

So, tshark decoder fails with some JSON error.

I suppose it's some encoding problem. Here is a similar issue, except Heplify removes more than just \x00.

[lmangani]

@mkpaz thanks for raising this can you provide a pcap to reproduce this end-to-end?

[mkpaz]

@lmangani Thanks for the response. Yes, I've attached all info including pcap for both SIP and HEP side.

isup_encoding_issue.zip

[mkpaz]

@lmangani Any suggestions would be greatly appreciated. I've tried to compile a look at it myself, but I'm not a Go coder.

[lmangani]

Thanks for your patience @mkpaz! Busy times. @adubovikov will review and patch if needed, once time allows

[mkpaz]

No longer interested. Feel free to reopen if the project is still maintained.

[lmangani]

The project is absolutely maintained, but sadly the resources are scarce. Apologies for letting this fall behind!

[adubovikov]

the problem is here in postgress, the data in ISUP part is full binary and the field type "varchar" doesn't support and rejects some "binary" elements. The best way to do it, change "raw - varchar" to "raw - bytea", but this will require to change also select/insert query. We will test it in the lab and let you know

[adubovikov]

so here is the way

postgres=# CREATE TABLE IF NOT EXISTS hep_proto_101_default (
                id BIGSERIAL NOT NULL,
                sid varchar NOT NULL,
                create_date timestamp with time zone DEFAULT CURRENT_TIMESTAMP NOT NULL,
                protocol_header jsonb NOT NULL,
                data_header jsonb NOT NULL,
                raw bytea NOT NULL
        );
CREATE TABLE
postgres=# \d
                          List of relations
 Schema |             Name             |       Type        |  Owner   
--------+------------------------------+-------------------+----------
 public | hep_proto_101_default        | table             | postgres
 public | hep_proto_101_default_id_seq | sequence          | postgres
(2 rows)

postgres=# INSERT INTO hep_proto_101_default (id, sid, create_date, protocol_header, data_header, raw) VALUES (1, 'aaa', '2014-06-12 20:36:50', '{}', '{}', 'aassdsdsddsfsdf');
INSERT 0 1
postgres=# INSERT INTO hep_proto_101_default (id, sid, create_date, protocol_header, data_header, raw) VALUES (1, 'aaa', '2014-06-12 20:36:50', '{}', '{}', 'bbbbbbasd2323'::bytea);
INSERT 0 1

postgres=# select * from hep_proto_101_default;
 id | sid |      create_date       | protocol_header | data_header |               raw                
----+-----+------------------------+-----------------+-------------+----------------------------------
  1 | aaa | 2014-06-12 20:36:50+02 | {}              | {}          | \x616173736473647364647366736466
  1 | aaa | 2014-06-12 20:36:50+02 | {}              | {}          | \x62626262626261736432333233
(2 rows)

postgres=# select * from hep_proto_101_default where raw LIKE '%bbbb%';
 id | sid |      create_date       | protocol_header | data_header |             raw              
----+-----+------------------------+-----------------+-------------+------------------------------
  1 | aaa | 2014-06-12 20:36:50+02 | {}              | {}          | \x62626262626261736432333233
(1 row)

postgres=# select * from hep_proto_101_default where raw LIKE '%aa%';
 id | sid |      create_date       | protocol_header | data_header |               raw                
----+-----+------------------------+-----------------+-------------+----------------------------------
  1 | aaa | 2014-06-12 20:36:50+02 | {}              | {}          | \x616173736473647364647366736466
(1 row)

postgres=# select encode(raw,'escape') from hep_proto_101_default where raw LIKE '%aa%';
     encode      
-----------------
 aassdsdsddsfsdf
(1 row)

postgres=# select encode(raw,'escape') from hep_proto_101_default where raw LIKE '%bb%';
    encode     
---------------
 bbbbbbasd2323
(1 row)

so, we should change it to bytea and in the select, we have to convert the raw into "hex" or to "escape" string.

@mkpaz sounds good for you ?