firehol
Incorrect action in update-ipsets "no need to load ipset in kernel"
I'm receiving a message saying there is no need to load ipset in kernel, this seems to be in error as the my ipsets are completely empty.
The issue appears to be at line 2426. $sets is empty/unset at that point... causing the test if [ -z "${sets[$ipset]}" ] to always resolve true.
The only place I could find where the associative array sets is modified is when it is initialized at line 933 and filled at 936. I don't see anything unsetting it... so I'm assuming a scoping issue of some sort.
I'm using bash 5.0.7 and firehol 3.1.6 .
modified code snippet:
...
ipset_apply() {
local ipset="${1}" ipv="${2}" hash="${3}" file="${4}" entries= tmpname= opts= ret= ips=
### ADDED FOR DEBUGING ###
echo "ipset: ${ipset}"
echo "sets: ${sets[@]}"
echo "current set: ${sets[$ipset]}"
### /ADDED FOR DEBUGING ###
if [ ${IPSETS_APPLY} -eq 0 ]
then
ipset_saved "${ipset}" "I am not allowed to talk to the kernel."
return 0
fi
ipset_apply_counter=$[ipset_apply_counter + 1]
tmpname="tmp-$$-${RANDOM}-${ipset_apply_counter}"
if [ "${ipv}" = "ipv4" ]
then
if [ -z "${sets[$ipset]}" ]
then
ipset_saved "${ipset}" "no need to load ipset in kernel"
# $IPSET_CMD --create ${ipset} "${hash}hash" || return 1
return 0
fi
...
Result:
sslproxies| 10/10 mins passed, downloading...
| fetch: 'http://www.sslproxies.org/'
| HTTP/200 OK
| downloaded successfully
| saving downloaded file
| source file has been updated
| converting with 'extract_ipv4_from_any_file'
ipset: sslproxies
sets:
current set:
| SAVED no need to load ipset in kernel
| version 3, 100 unique IPs
|
sslproxies_1d| merging history files (1440 mins)
ipset: sslproxies_1d
sets:
current set:
| SAVED no need to load ipset in kernel
| version 3, 260 unique IPs
|
sslproxies_7d| merging history files (10080 mins)
ipset: sslproxies_7d
sets:
current set:
| SAVED no need to load ipset in kernel
| version 3, 260 unique IPs
|
sslproxies_30d| merging history files (43200 mins)
ipset: sslproxies_30d
sets:
current set:
| SAVED no need to load ipset in kernel
| version 3, 260 unique IPs
Cannot comment on the exact cause, but I get the same symptom (update-ipsets says no need to load ipset in kernel, but ipset list -n returns 0 ipsets)
Looks like sets is populated via ipset_list_names which runs ipset --list -t, but if there are no ipsets created yet, then ipset_apply will return if no sets= are defined.
Am I missing something, or is the initial creation of ipsets broken?
It is August 2021 and as a new user, I am still encountering this same exact issue... @ktsaou ... anything to improve here? Suspect @ktsaou has gotten tired of supporting this script or may be he is no longer able to do it for reasons unknown.
But I found that in 2017 he acknowledged that update-ipsets doesn't work if you don't first create the lists in ipset, here is what he said regaring this same issue in another still open issue. Thus my statement that we can't count on the old maintaners authors to keep on supporting this script.
"Hi,
you can use ipset directly from firehol.conf. Check this: https://github.com/firehol/firehol/wiki/Working-with-IPSETs
If you don't user firehol, you can use ipset-apply.sh from the contrib directory of firehol: https://github.com/firehol/firehol/blob/master/contrib/ipset-apply.sh This script takes an iprange/update-ipsets compatible input file (.ipset or .netset) and loads it into kernel.
Once an ipset is loaded into the kernel update-ipsets will update it automatically."
I can report that this works like a charm, basically the trick is that the lists MUST exist in ipset before running update-ipsets! How you create them, doesn't matter. probably to get around the initialization bug mentioned by @JadedDragoon in 2019.
After executing sudo update-iptables (after having executed the sudo update-iptables enable xxxx), the .ipsets and .netsets files will be downloaded and created in your system. You can find them with this command:
sudo find /etc -name *.ipset -type f sudo find /etc -name *.netset -type f
then find the ipset-apply.sh script in your system and execute this command for each file you found above for your lists to block (path is for Ubunty 20.04)
sudo /usr/share/doc/firehol/contrib/ipset-apply.sh /etc/firehol/ipsets/blocklist_de.ipset
I didn't test if adding only one ipset is sufficient to get around the bug or not, I just added all the ones I wanted to ipset and then run update-ipsets.
But I incorrectly assumed that update-ipsets was going to add the lists to iptables as part of the script, but it doesn't, you have to add them one by one to the rules using:
sudo iptables -I INPUT -m set --match-set blocklist_de src -j DROP
this will block all ports for the ips on the list. for specific protocols you need to add the -p option.
Before leaving this as done, I'm checking the ipset-apply.sh and just discovered that they use the ipset -N option for create, but that option is no longer on the man pages of the ipset Debian web page. I politely suggest to the mantainers to change this line in the script and replace -N for create (or n as the shortcut, no dash ahead of it). Change -R (that used to be the restore command) for restore -F for flush and -W for swap (or w as the shortcut).
This is the core of what this script does:
echo >&2 "Creating a temporary ipset..."
ipset "create" "${tmpname}" ${hash}hash ${opts} || exit 1
echo >&2 "Flushing the temporary ipset..."
ipset "flush" "${tmpname}" || exit 1
echo >&2 "Loading the temporary ipset with the IPs in file ${file}..."
ipset "restore" <"/tmp/${tmpname}" || exit 1
