node-libcurl icon indicating copy to clipboard operation
node-libcurl copied to clipboard

Reuse `ticket` in extension `pre_shared_key`

Open iamoskvin opened this issue 1 year ago • 7 comments

I was not able to see this extension in a TLS Hello. Is it supported by this lib?

iamoskvin avatar Oct 21 '24 09:10 iamoskvin

it's support, it will be in the extension after the first request

Ossianaa avatar Oct 21 '24 09:10 Ossianaa

it's support, it will be in the extension after the first request

I have tried this already but without success. What am I doing wrong? I also checked in wireshark, the pre_shared_key is absent.

`import { LibCurl, fetch, requests } from "@ossiana/node-libcurl"; import { LibCurlHttpVersionInfo } from "@ossiana/node-libcurl/dist/libcurl.js";

try { const resp = await fetch("https://tools.scrapfly.io/api/fp/ja3?extended=1", { httpVersion: LibCurlHttpVersionInfo.http2, // openInnerLog: true, connectReuse: false, }); const body = await resp.json(); console.log(body.tls.extensions);

const resp2 = await fetch("https://tools.scrapfly.io/api/fp/ja3?extended=1", { httpVersion: LibCurlHttpVersionInfo.http2, // openInnerLog: true, connectReuse: false, }); const body2 = await resp2.json(); console.log(body2.tls.extensions); } catch (e) { console.log(e); }`

I got this both times ['GREASE (0xEAEA)', 'application_layer_protocol_negotiation (16) (IANA)', 'compress_certificate (27) (IANA)', 'key_share (51) (IANA)', 'signed_certificate_timestamp (18) (IANA)', 'status_request (5) (IANA)', 'extended_master_secret (23) (IANA)', 'extensionEncryptedClientHello (65037) (boringssl)', 'extensionRenegotiationInfo (boringssl) (65281) (IANA)', 'supported_versions (43) (IANA)', 'extensionApplicationSettings (17513) (boringssl)', 'supported_groups (10) (IANA)', 'session_ticket (35) (IANA)', 'server_name (0) (IANA)', 'psk_key_exchange_modes (45) (IANA)', 'ec_point_formats (11) (IANA)', 'signature_algorithms (13) (IANA)', 'GREASE (0x3A3A)']

iamoskvin avatar Oct 21 '24 12:10 iamoskvin

use a same instance like this code

(async () => {
    const session = requests.session();
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
})();

Ossianaa avatar Oct 21 '24 12:10 Ossianaa

use a same instance like this code

(async () => {
    const session = requests.session();
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
})();

Thanks! It worked.

  1. What are the benefits of using sessions vs fetch? The easy cookie management or something else?
  2. Is it possible to somehow serialize the preshared_key, save in a file and then restore the tls session after the program restart? I understand that it is currently possible to just preserve the libcurl instances, but the idea is more long-term session storing between a nodejs script runs.

iamoskvin avatar Oct 21 '24 13:10 iamoskvin

use a same instance like this code

(async () => {
    const session = requests.session();
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
    console.log(
        await session
            .get("https://tools.scrapfly.io/api/fp/ja3?extended=1", {})
            .then((e) => e.text),
    );
})();

Thanks! It worked.

  1. What are the benefits of using sessions vs fetch? The easy cookie management or something else?
  2. Is it possible to somehow serialize the preshared_key, save in a file and then restore the tls session after the program restart? I understand that it is currently possible to just preserve the libcurl instances, but the idea is more long-term session storing between a nodejs script runs.
  1. use requests easy manage curl_impl instance, I'm usually used to writing this way. fetch is simply easier for front-end developers to use.
  2. It seems achievable, but to what end are you doing it

Ossianaa avatar Oct 22 '24 16:10 Ossianaa

2. It seems achievable, but to what end are you doing it Thanks! The goal is to better mimic real browser behavior in cases when it is needed to run one or more scripts several times. For example, I am saving all needed headers (user-agent etc), so I can use them in every script run. But with a preshared_key it is impossible, so every new run is starting the TLS handshake from the beginning and it does not look like real Chrome behavior? Does it make sense?

iamoskvin avatar Oct 22 '24 16:10 iamoskvin

  1. It seems achievable, but to what end are you doing it Thanks! The goal is to better mimic real browser behavior in cases when it is needed to run one or more scripts several times. For example, I am saving all needed headers (user-agent etc), so I can use them in every script run. But with a preshared_key it is impossible, so every new run is starting the TLS handshake from the beginning and it does not look like real Chrome behavior? Does it make sense?

Seems good, maybe in the next version

Ossianaa avatar Oct 22 '24 17:10 Ossianaa