Why this server publish more certs than the CaliDog's

[Baoxd123]

Firstly, thanks so much for this repo so we can replace the CaliDog's certstream(which is down)!

However, I found this certstream server generates more certificates than CaliDog's per second, which makes me confused. Do you have any ideas about this phenomenon?

[d-Rickyy-b]

Hi there, interesting question. Can you provide more information about the certificates you receive additionally from the certstream-server-go compared to the official certstream server?

The one huge difference I know of is that the official server uses the full list of all known logs while certstream-server-go uses only the logs used in Google Chrome.

• Official: https://github.com/CaliDog/certstream-server/blob/60de7000901e5eb246d2e83c908678b43e5a60c8/lib/certstream/ct_watcher.ex#L25C21-L25C78
• certstream-server-go: https://github.com/d-Rickyy-b/certstream-server-go/blob/2b7f5060b5348cb8acaf6ae32e20d727ee492130/internal/certificatetransparency/ct-watcher.go#L312

Maybe the hosted certstream server by calidog was overloaded? There are tons of people connecting directly to the calidog server, despite it just being there for demo purposes. It's recommended even by the author to host your own certstream server. In the past, he ran into some bandwidth limitations: https://x.com/fitblip/status/1552162750977171458

[Baoxd123]

Thanks for your inspiring reply. Yes, I think the overloading is an important reason. According to my local log, the CaliDog's speed for publishing certs was usually around 100 certs/sec but your server's speed which is around 300 certs/sec should be the correct speed.

By the way, I observed that the collection speed of certstream would also reduce if my program machine was overloading, even if the certstream-go server was deployed on another machine.

[Knight1]

I had the same problem. With calidog i used a small vm to process all certs. Now i have to use two big vms.

The only idea i had was that calidog might use deduplication.

[d-Rickyy-b]

The only idea i had was that calidog might use deduplication.

I can't remember seeing deduplication in their code. It would also mean to store certificates or at least their IDs in order to deduplicate them so that when they are shared via other logs some times later you can identify them.

Anyway, I think this question has been answered, so I'll close the issue. Feel free to open another one, or start a discussion in the Discussions tab.