tldextract

tldextract copied to clipboard

setting PUBLIC_SUFFIX_LIST_URLS with environment variable

5

`PUBLIC_SUFFIX_LIST_URLS` now can only define with the function arguments, can this define by environment?

x0day

IPv6 addresses are not handled

6

For URLs using IPv4 addresses, the host address gets extracted correctly using `.domain` but `.fqdn` gives the empty string: ```python >>> tldextract.extract("https://127.0.0.1:1234/foobar").domain '127.0.0.1' >>> tldextract.extract("https://127.0.0.1:1234/foobar").fqdn '' ``` For URLs using...

ohad-ivix

How can I do 1.2 billion extractions faster?

15

I recently used this library to extract the second-level domain names from the 1.2 billion PTR records in Rapid7's [Sonar](https://opendata.rapid7.com/sonar.rdns_v2/) database. As an example it would extract ``totinternet`` from ``node-nnf.pool-1-1.dynamic.totinternet.net``....

marklit

Timeout: The file lock 'some/path/to/8738.tldextract.json.lock' could not be acquired

5

I start getting this error when I increase the number of processes / threads to a certain point. Is there a way to increase the timeout value? More importantly, why...

rudolfovic

requests.Session pass-in (for customizing things to do with the PSL fetch)

1

in `tldextract/remote.py` it should allow passing in a custom instance of `Session` from `requests`. need that change and then changes up the call-stack so that this custom session can be...

floer32

Reconsider network auto-update by default

5

While it's understandable and useful in many situations to want the latest dataset, it can cause issues in some situations: - ephemeral environments that will not be able to cache...

brycedrennan

Deprecate or remove use __file__

5

For reading `.tld_set`, `__file__` should be replaced with pkgutil.get_data, which is used for `.tld_set_snapshot`. The dataset file may have been placed into the package data via another means, and the...

jayvdb

Reconsider cache folder

4

Reconsider caching in the library's install folder. The GitHub issue tracker is rife with confusion about the permission warning (#9), or outright uncaught exceptions (#209). Finally do something about it....

john-kurkowski

Question: how to exclude a TLD to deem it as not allowed/valid?

1

How can we exclude a tld from the PSLs loaded by the library, is there a setting for that?

ambersariya

Allow cache_file argument to also accept list

8

Problem: We have an application, which perform TLD operation using celery workers. As we have a couple of celery workers, whenever cache_file update is called, it only updates the file...

hibare