Slow browsing in windows 10

[yuvalpikel]

Hi , we used the demo code from the Readme file but the browsing is extremely slow What do we miss ?

Regards Yuval

[honfika]

what does it mean exctremly slow? Minutes to load a page?

Is it faster when you laod the page 2nd time?

HTTPS or HTTP?

[yuvalpikel]

Hi

Seconds .. but 30 - 40 % slower

We are using HTTPS

On Thu, Dec 26, 2019 at 11:41 AM honfika [email protected] wrote:

what does it mean exctremly slow? Minutes to load a page?

Is it faster when you laod the page 2nd time?

HTTPS or HTTP?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/justcoding121/Titanium-Web-Proxy/issues/719?email_source=notifications&email_token=AHWDTATDZQNFC4QDRCLGVC3Q2R34RA5CNFSM4J7J42W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHVKFAQ#issuecomment-569025154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWDTAWVV2SCWBVX3FJHVQTQ2R34RANCNFSM4J7J42WQ .

[honfika]

Only the first time or always? I would not say that 30%-40% is "extremly slow" At first time TWP should create a certificate which can be a little bit slow.

[yuvalpikel]

Always... The idea was to monitor the traffic , we can not slow the users so much

Is there any parameter that can enhance the performance?

On Thu, Dec 26, 2019 at 12:04 PM honfika [email protected] wrote:

Only the first time or always? I would not say that 30%-40% is "extremly slow"

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/justcoding121/Titanium-Web-Proxy/issues/719?email_source=notifications&email_token=AHWDTASM7S2VNVPFAASR3YLQ2R6SFA5CNFSM4J7J42W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHVLECA#issuecomment-569029128, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWDTATGIS4X6EU3MEOQJ4DQ2R6SFANCNFSM4J7J42WQ .

[honfika]

Could you please provide a small project which is slow? The readme file contains a lot of code which coudl make it slower...

for example reading the request/response to the memory then sending to the client: byte[] bodyBytes = await e.GetRequestBody();

You should not do that if you don't need to read/modify the body

[yuvalpikel]

I have to read the body ..of a requests from a particular site

[yuvalpikel]

This is my code :

var proxyServer = new ProxyServer();

proxyServer.BeforeRequest +=  ProxyServer_BeforeRequest;
proxyServer.AfterResponse += ProxyServer_AfterResponse;

var explicitEndPoint = new ExplicitProxyEndPoint(IPAddress.Any, 8000, true)
{
};

proxyServer.AddEndPoint(explicitEndPoint);
proxyServer.Start();

foreach (var endPoint in proxyServer.ProxyEndPoints)
    Console.WriteLine("Listening on '{0}' endpoint at Ip {1} and port: {2} ",
        endPoint.GetType().Name, endPoint.IpAddress, endPoint.Port);

// Only explicit proxies can be set as system proxy!
            
proxyServer.SetAsSystemHttpsProxy(explicitEndPoint);

// wait here (You can use something else as a wait function, I am using this as a demo)

Console.Read();
            
proxyServer.BeforeRequest -= ProxyServer_BeforeRequest;            
proxyServer.AfterResponse -= ProxyServer_AfterResponse;
            
proxyServer.Stop();

[honfika]

And you read that body only? IS that a big page? How big? Is that a public site? How do you measure the speed?

Normally a - average - webpage loads in 1-2 seconds... 1,5-3 seconds does not seem to me extremly slow.

[honfika]

Ok, but the "real" slow code shoud be in ProxyServer_BeforeRequest and ProxyServer_AfterRequest

And you mentioned Windows 10... is it faster on earlier windows or on other system?

[yuvalpikel]

Is the code i've sent is OK ?

[honfika]

Yes, but I need the content of the event handlers too.

[honfika]

And what framework do you use? .NET Core version (netstandard 2.1) should be faster

[yuvalpikel]

private static  async Task ProxyServer_BeforeRequest(object sender, SessionEventArgs e)
{
    if (e.HttpClient.Request.Url.ToLower().Contains("Strauss.com"))
    {
        var method = e.HttpClient.Request.Method.ToUpper();

        string bodyString = await e.GetRequestBodyAsString();
        exPackage curPack = new exPackage { StartRequest = DateTime.Now, Method = method, Body = bodyString , URL = e.HttpClient.Request.Url };
        e.UserData = curPack;
    }
}
      

[yuvalpikel]

private static    Task ProxyServer_AfterResponse(object sender, SessionEventArgs e)
{
    string method = e.HttpClient.Request.Method;

    if (e.UserData != null)
    {
        exPackage curPack = (exPackage)e.UserData;
        sessions.Enqueue(curPack);
    }

    return Task.FromResult(0);
}

[yuvalpikel]

We are using .NET 4.61

[honfika]

Is it possible to upgrade to .net core?

[honfika]

By the way, I don't think that this condition will be true any time:

if (e.HttpClient.Request.Url.ToLower().Contains("Strauss.com"))

Since it contains an upper cased S

[yuvalpikel]

I will upgrade .. Thanks for the S note :)

[honfika]

Ok, please test it with .net core, and let me know if it is still slow.

[honfika]

If you want to decrypt only 1 (or some) hosts, I suggest you to disable ssl decryption for other sites:

explicitEndPoint.BeforeTunnelConnectRequest += ExplicitEndPoint_BeforeTunnelConnectRequest;

private static Task ExplicitEndPoint_BeforeTunnelConnectRequest(object sender, TunnelConnectSessionEventArgs e)
{
    e.DecryptSsl = CultureInfo.CurrentCulture.CompareInfo.IndexOf(e.HttpClient.Request.Url, "strauss.de", CompareOptions.IgnoreCase) >= 0;
    return Task.CompletedTask;
}

[honfika]

This is my measures with a big hungarian new site:

Without proxy:

With proxy

[yuvalpikel]

Thanks .. any other recommendations to improve performance?

[honfika]

Do you use release build?:)

[honfika]

NET 4.6.1 Release build:

NET Core Release build:

[honfika]

Could you please compare the performace with Fidder? (https://www.telerik.com/fiddler)

[ByronAP]

Decryption on widows is slow af compared to linux... but i think everyone already knows that. Not sure if fiddler is using some custom lib to do decryption but it is faster.

[ByronAP]

measurements on my end show an average of about 400ms added to each request when going through TWP (win 10, .net core, TWP latest with a bit of overhead due to auth, etc..) test url https://echo.proxy.services/

[yuvalpikel]

Are you sure ? How much faster?

בתאריך יום ו׳, 27 בדצמ' 2019, 3:04, מאת Allen Byron Penner ‏< [email protected]>:

Decryption on widows is slow af compared to linux... but i think everyone already knows that. Not sure if fiddler is using some custom lib to do decryption but it is faster.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/justcoding121/Titanium-Web-Proxy/issues/719?email_source=notifications&email_token=AHWDTAREKXKUPRAR36GQOU3Q2VIB5A5CNFSM4J7J42W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHWK22A#issuecomment-569159016, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWDTAXTJVL2EESBSZAWTF3Q2VIB5ANCNFSM4J7J42WQ .

[yuvalpikel]

And in fiddler?

בתאריך יום ו׳, 27 בדצמ' 2019, 3:07, מאת Allen Byron Penner ‏< [email protected]>:

measurements on my end show an average of about 400ms added to each request when going through TWP (win 10, .net core, TWP latest with a bit of overhead due to auth, etc..)

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/justcoding121/Titanium-Web-Proxy/issues/719?email_source=notifications&email_token=AHWDTAWZAQ3YK5XFBFUZDVTQ2VILZA5CNFSM4J7J42W2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHWK4VQ#issuecomment-569159254, or unsubscribe https://github.com/notifications/unsubscribe-auth/AHWDTARWFBW7BIFLLNMGCZDQ2VILZANCNFSM4J7J42WQ .

[honfika]

@ByronAP How do you emaseure this latecy? I created a test code, and measured that the difference is only about 3-8 ms/request

Code:

        static void Main(string[] args)
        {
            var proxyServer = new ProxyServer();

            proxyServer.BeforeRequest += ProxyServer_BeforeRequest;
            proxyServer.AfterResponse += ProxyServer_AfterResponse;

            var explicitEndPoint = new ExplicitProxyEndPoint(IPAddress.Any, 8000, true)
            {
            };

            proxyServer.AddEndPoint(explicitEndPoint);

            proxyServer.EnableConnectionPool = false;
            proxyServer.Start();

            foreach (var endPoint in proxyServer.ProxyEndPoints)
                Console.WriteLine("Listening on '{0}' endpoint at Ip {1} and port: {2} ",
                    endPoint.GetType().Name, endPoint.IpAddress, endPoint.Port);

            const int count = 10;
            string url = "https://echo.proxy.services/";
            var wc = new WebClient();
            wc.DownloadData(url);

            double totalTime = 0;
            for (int i = 0; i < count; i++)
            {
                var sw = Stopwatch.StartNew();
                wc.DownloadData(url);
                totalTime += sw.ElapsedMilliseconds;
            }

            Console.WriteLine($"No proxy avg: {(totalTime / count)}ms");

            wc = new WebClient();
            wc.Proxy = new WebProxy("127.0.0.1:8000");
            wc.DownloadData(url);

            totalTime = 0;
            for (int i = 0; i < count; i++)
            {
                var sw = Stopwatch.StartNew();
                wc.DownloadData(url);
                totalTime += sw.ElapsedMilliseconds;
            }

            Console.WriteLine($"Proxy avg: {(totalTime / count)}ms");

            Console.Read();

            proxyServer.BeforeRequest -= ProxyServer_BeforeRequest;
            proxyServer.AfterResponse -= ProxyServer_AfterResponse;
            proxyServer.Stop();
        }

        private static async Task ProxyServer_BeforeRequest(object sender, SessionEventArgs e)
        {
            string str = e.HttpClient.Request.Url;
            Console.WriteLine(e.HttpClient.Request.Url);
        }

        private static Task ProxyServer_AfterResponse(object sender, SessionEventArgs e)
        {
            return Task.CompletedTask;
        }
    }

Output:

No proxy avg: 354,1ms
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
https://echo.proxy.services/
Proxy avg: 356,9ms

And when I set the connection pooling enabled (which is the default), the result is 2x faster with the proxy. (About 150ms/rq)

[honfika]

Please try to set the following parameters after the ProxyServer constructor call: proxyServer.CertificateManager.CertificateEngine = CertificateEngine.BouncyCastleFast; proxyServer.EnableConnectionPool = false; proxyServer.ThreadPoolWorkerThread = 64;

(the first setting needs the latest TWP version)

[ByronAP]

I tested haproxy -> TWP -> host then I test haproxy -> host

I log to influxdb and can see the timings between haproxy and host which showed on average +300 ms when going through TWP

... sorry for the late reply i didn't get an alert ...