Testing Docker

[mgrimace]

Hello, I figured I'd contribute by testing out the docker implementation. tl;dr - Initial run has no issues(!), testing scheduled runs next.

Here's the steps I've taken in case anyone more knowledgable can test/contribute/make suggestions.

System

Headless Debian 12 server, running on a Proxmox LXC

Steps

• Downloaded the source code
• Updated accounts.json
• Made sure config.json is set to "headless": true,
• Per #29 I also changed "searchDelay": { to "min": 180000, and "max": 270000 because I'm in a region with a cooldown.

Build and run the docker image

• Run docker build -t microsoft-rewards-script-docker . to build the image
• docker run --name netsky -d microsoft-rewards-script-docker omit the detached flag -d to view the script output in terminal. Optionally, change the name of the container.

Schedule daily runs

This is where I'm less certain as to best method to do this, presently:

• Create a daily.sh script, which should add a degree of randomness to the start time:

#!/bin/bash

MINWAIT=$((5*60))
MAXWAIT=$((50*60))
SLEEPTIME=$((MINWAIT+RANDOM % (MAXWAIT-MINWAIT)))
echo "sleeping for "$((SLEEPTIME / 60))"m ("$SLEEPTIME"s) ..." 
sleep $SLEEPTIME
echo "starting container"
docker start netsky

• Make sure the script is executable
• Set up cron to run the script daily using crontab -e and entering the following line, which should run the script at 5:30 am daily + the random sleeptime calculation above:

30 5 * * * /bin/bash /root/Microsoft-Rewards-Script/daily.sh

Results

I'll report back with results here. I'm more comfortable using docker compose, so this is new territory for me. Suggestions welcome.

[mgrimace]

Initial manual run of the script completed without error. I had already completed searches, etc. earlier in the day, so I can't say for certain if those would have worked, but no errors getting it up and running!

[mgrimace]

Ok, scheduled run started as expected. Desktop searches completed, and mobile started but ran into an error approximately half-way through:

[3/28/2024, 1:45:26 PM] [PID: 31] [LOG] [MAIN] Unable to complete mobile searches, bad User-Agent? Retrying...

It retried 5 times, with errors relating to timeout exceeded (I can provide additional logs, not sure what’s helpful), then stopped.

Edit: second run of the container completed the remaining searches, which, for me is more-or-less typical behaviour (I ran the base script 2x daily previously in case anything didnt’ finish)

[mgrimace]

Testing run via an early/rough compose.yaml with built-in scheduling, and definitely needs testing and revision. The goal is to keep the container running while the script itself is re-run daily per the in-built cron schedule and script. There is very likely a smarter/more efficient way to do this.

The following assumes you have a script called run_daily.sh located in the root of the package (/Microsoft-Rewards-Script/run_daily.sh). I'm testing out the following script, which is slightly changed from above to re-run the rewards script rather than restart the container.

run_daily.sh

#!/bin/bash

# Define the minimum and maximum wait times in seconds
MINWAIT=$((5*60))  # 5 minutes
MAXWAIT=$((50*60)) # 50 minutes

# Calculate a random sleep time within the specified range
SLEEPTIME=$((MINWAIT + RANDOM % (MAXWAIT - MINWAIT)))

# Convert the sleep time to minutes for logging
SLEEP_MINUTES=$((SLEEPTIME / 60))

# Log the sleep duration
echo "Sleeping for $SLEEP_MINUTES minutes ($SLEEPTIME seconds)..."

# Sleep for the calculated time
sleep $SLEEPTIME

# Log the start of the script
echo "Starting script..."

# Start the script
docker exec netsky npm run start

compose.yaml

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - TZ=America/Toronto #change to your timezone
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #make sure this matches where you've installed the package
    restart: unless-stopped
  cron:
    image: alpine
    container_name: cron-container
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #should be same as above
    restart: unless-stopped
    command: ["sh", "-c", "echo '30 5,11 * * * root /usr/src/microsoft-rewards-script/run_daily.sh' > /etc/crontabs/root && crond -f"] # this runs script daily at 5:30 am and 11:30 am of your timezone using the added randomness from the run_daily.sh, feel free to change

[mgrimace]

Testing run via an early/rough compose.yaml with built-in scheduling, and definitely needs testing and revision. The goal is to keep the container running while the script itself is re-run daily per the in-built cron schedule and script. There is very likely a smarter/more efficient way to do this.

This setup won't work. The rewards script exits after running, then immediately restarts, so it'll run constantly. Not sure how to fix at the moment, but I'll keep testing. The docker build + docker run setup work fine in the meantime

[Ryely-Squires]

Following. I'm absolutely horrid with regular docker, I only use compose, and throwing this bot in a container would be nice.

[mgrimace]

Following. I'm absolutely horrid with regular docker, I only use compose, and throwing this bot in a container would be nice.

Ha me too! If you want test a compose, please try the one below. AFAIK, you'd want to place this as a compose.yaml the root of the Microsoft-Rewards-Script folder, and run with docker compose up -d or actually omit the -d to watch if it works.

If you have previously used the docker run commands, you may have to use docker stop netsky and docker rm netsky to remove the previously ran version.

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - TZ=America/Toronto #customize, may not be necessary at the moment 
    volumes:
      - /path/to/Microsoft-Rewards-Script:/usr/src/microsoft-rewards-script #customize this to wherever you've installed the package

This should run once, then exit. For now, if you want to schedule it, copy the run_daily.sh below into the same folder, and add a cron entry using crontab -e and add the following line 30 5 * * * /bin/bash /path/to/Microsoft-Rewards-Script/daily.sh (making sure to actually use the real path, not the placeholder I have here). It's set for 5:30 am, if you want it to run twice, just do a comma after the 5 (e.g., 30 5,13 * * * for 5:30 am and 1:30 pm, or customize as you wish).

run_daily.sh

#!/bin/bash

MINWAIT=$((5*60))
MAXWAIT=$((50*60))
SLEEPTIME=$((MINWAIT+RANDOM % (MAXWAIT-MINWAIT)))
echo "sleeping for "$((SLEEPTIME / 60))"m ("$SLEEPTIME"s) ..." 
sleep $SLEEPTIME
echo "starting container"
docker start netsky

Finally, if you've been messing around like I have and building these left and right, you might want to run a docker system prune -a -f to cleanup.

[mgrimace]

The next goal would be to add environmental variable support for things like headless=true directly into the compose, and support for environmental secrets (.env) for account information, but that requires updating the dockerfile, configs, and possible other things that are outside my basic skills!

[Ryely-Squires]

I've been messing with the compose for a little while now, and I can get it to actually startup (along with the webhook!) but have been getting errors within the program itself. I'm sure they're on my end in terms of my setup (I recently transferred from GUI to CLI Debian, everything is a learning curve). But I can confirm that the compose works well, and shows logs! If I can get it fully working, I'll let you know. Great work.

[mgrimace]

I've been messing with the compose for a little while now, and I can get it to actually startup (along with the webhook!) but have been getting errors within the program itself. I'm sure they're on my end in terms of my setup (I recently transferred from GUI to CLI Debian, everything is a learning curve). But I can confirm that the compose works well, and shows logs! If I can get it fully working, I'll let you know. Great work.

Thanks for testing! Here is a simplified compose that should be run from the root directory:

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
    volumes:
      - .:/usr/src/microsoft-rewards-script

AFAIK, sending logs/results to Discord via webhooks should not require incoming ports to be exposed and it should work out-of-the-box. Can't speak to other errors though, but check my other post #29 if searches are failing (due to cooldown requirements in some countries)

[Ryely-Squires]

The error was a network one (net::ERR_NETWORK_CHANGED) which seems to have gone away with redoing the setup process? I'm not sure, but the bot has ran successfully once.

I haven't been able to test whether it can correctly search for points yet because I have already received them for today, but I've put the shell script and edited cron accordingly. I will again update you tomorrow on whether or not it successfully runs automatically with the scripts.

[Ryely-Squires]

The script seems to work.

I should mention that the dreaded network error seems to have reared it's head, but whether or not it'll happen seems to be random with each restart? I'm not sure, I'll likely make a separate issue on the matter if I can't figure it out. I had to run the shell script manually a few times (with shortened durations) to get the container to actually function. And looking at the cronjob, it should be fine.

[mgrimace]

The script seems to work.

I should mention that the dreaded network error seems to have reared it's head, but whether or not it'll happen seems to be random with each restart? I'm not sure, I'll likely make a separate issue on the matter if I can't figure it out. I had to run the shell script manually a few times (with shortened durations) to get the container to actually function. And looking at the cronjob, it should be fine.

It’s possible it’s a docker-related networking error, try docker network prune if you’ve been playing around in docker. Here’s a similar issue on Stackoverflow (link)

[Ryely-Squires]

I have tried that command a few times in testing, I thought maybe because I had overdone it on creating containers/networks, but it seems to be the same. Regardless, running it a few times basically guarantees that at least one of them will function, so I have the cronjob entry modified for a few times rather than one just to account for that. Otherwise, the setup seems to work well.

[mgrimace]

Ok, if you need to rebuild the script after making changes to the config, you'll need to run docker compose up -d --build to rebuild or else it'll just re-up the existing/previous build.

If you've re-built the image a few times now, you'll also like want to run docker system prune -a -f to clean up all the builds/images.

[mgrimace]

I'm currently testing automatic scheduled daily runs via docker. If you can contribute or test, please take a look at my docker test branch: https://github.com/mgrimace/Microsoft-Rewards-Script/tree/docker

It should run the script on the initial container start; then, it'll keep the container idle, and re-run the script daily at 5:00 am.

The schedule is customizable in the crontab file found under /src/. use crontab.guru if you're not sure how to change cron syntax.

If it works, ideally the next step would be to be able to customize the value via an environments: argument in the docker compose file.

If you are able to help or contribute to docker implementation in any way, please feel free to add a PR to my testing branch (and once everything is confirmed working, I'll PR it to main here).

[Ryely-Squires]

Hey, SSH into my server isn't working, so if nobody follows up before I head over there for the weekend, I'd be happy to test it out (although the current cronjob implementation works just fine). This seems a lot more user friendly.

[mgrimace]

Hey, SSH into my server isn't working, so if nobody follows up before I head over there for the weekend, I'd be happy to test it out (although the current cronjob implementation works just fine). This seems a lot more user friendly.

Thanks so much, and no rush, I'm kind of learning as I go with this. My overall intention with docker is to have everything 'built in' to the service and not require any coding/scripting (because that's something I struggle with myself).

Out of curiosity, do you think the default implementation should be one run per day (e.g., 5 am)? Or two runs (e.g., 5 am, 11am) to catch anything that was missed? I run it twice myself

[Ryely-Squires]

Almost certainly twice per day; I even run it three times as a precaution. The script tends to miss things on the first run.

[mgrimace]

Almost certainly twice per day; I even run it three times as a precaution. The script tends to miss things on the first run.

Great ok! I'm testing an update at the moment that adds custom cron schedules that can be set via environmental variables in compose, or in the config.json. It is not yet live on my docker testing branch (I want to make sure in-built scheduling actually works before pushing this update, which could make things messier)

With environmental variable support, the default value is 0 5,11 * * * which is pre-set in the config.json, but any changes made in the compose file should take precedence. For example, you could run it three times daily with:

services:
  microsoft-rewards-script:
    build: .
    container_name: netsky
    environment:
      - NODE_ENV=production
      - CRON_START_TIME=0 5,11,16 * * *   #Set your preferred schedule here, use crontab.guru for proper cron syntax 
    volumes:
      - .:/usr/src/microsoft-rewards-script

If a user doesn't set it in the compose, it should still default to 5,11; however, I'm attempting to make sure that if cron is removed from the compose AND from the config.json that the script only runs on container launch (which is the base functionality). Testing it for a 24 hr cycle then I'll push that update.

Upcoming goal is to set other env variables like headless mode, search delays, etc. that can all be done in the compose.

[mgrimace]

I should mention this is not yet live in my docker branch. It only has the in-built scheduling, not the env variable support yet (I want to test it further before breaking the current in-built scheduling).

If you or anyone do test the in-built scheduling, remember to comment out your exising crontab entries by using crontab -e and andding a # before you current schedule. The container should schedule and run itself, and look for it to run the script on the next schedule (5am) automatically.

You can use docker logs netsky to see what the script is doing in the container.

[mgrimace]

Ok, I pushed a huge update to my docker testing branch.

This update includes substantial changes to the DOCKERFILE to enable docker environmental variable support for all config options. Absolutely no changes are made to the core .ts files or underlying script.

I added each env var to the config table on the main README.md. HEADLESS=true is the only mandatory one for docker.

Scheduling:

• You can use the var - CRON_START_TIME= in run or compose to set your own preferred schedule. The script will run automatically on container start, then stay idle until the next schedule run.
• Scheduling is built into the container and requires no user scripting or system cron entries, it is all done via env var in run or compose. If the user does not set a CRON_START_TIME, it will default to 5 am and 11 am local time.
• Remember to disable or comment out any previous cron schedules on your system via crontab -e.

Set config options via environment vars:

• All config options have corresponding env vars that can be used in compose or run.
• I added Timezone support for the container
• If specified, the custom env var value will be prioritized over the default values present in the config.json. As in, you do not need to change the values in the config.json if you set a custom env var. For example, if you have HEADLESS=false in the config.json (default value), but set it as =true in the compose.yaml, it will run as HEADLESS=true.
• If you do not specify a particular env var, the script will automatically use the default value present in the config.json.

I'm learning as I go and doing this in my spare time. There will likely be bugs and things I've missed. Please be patient, and test and contribute wherever possible. PRs are welcome and encouraged.

https://github.com/mgrimace/Microsoft-Rewards-Script/tree/docker

I'll PR this back to main here after preliminary testing.

[mgrimace]

Ok, tl;dr - the docker container runs the script daily as scheduled by the user in compose/run command. It should have full support for env variables, which are prioritized over the config. This means, you can (hopefully) do everything except your account from a compose file or run command.

I removed running the script on container start because it was getting impossible to test. It only runs as scheduled with a default of 5 and 11, customizable by CRON_START_TIME. Kept script which adds randomness to scheduled start times as a preventative measure.

I've pushed a few updates to my docker testing branch to fix scheduling and timezone.

Next steps - account support for docker compose.yaml, maybe a 'run-once' env option, and exploring apprise support for better notifications (but this is tricky for me because multiple .ts files would need edititing). Further testing would be appreciated, PRs welcome. AFAIK this should be safe to push back here to main but I'm going to test a little more.

[Ryely-Squires]

Hey, sorry for taking so long.

I couldn't get it to run. The container on run shows timestamp of a time it currently isn't, and does so immediately (without the delay) going up to "Password entered successfully", and nothing after.

I've been troubleshooting for a while now, can't seem to fix it. My only guess is that it's conflicting with the original branch's installation, considering the time it says it's running at (1 AM) is what the main branch ran at.

I'm sure it'd probably work on a clean installation.

[mgrimace]

Hey, sorry for taking so long.

I couldn't get it to run. The container on run shows timestamp of a time it currently isn't, and does so immediately (without the delay) going up to "Password entered successfully", and nothing after.

I've been troubleshooting for a while now, can't seem to fix it. My only guess is that it's conflicting with the original branch's installation, considering the time it says it's running at (1 AM) is what the main branch ran at.

I'm sure it'd probably work on a clean installation.

No need to apologize at all, I appreciate you trying it out! I pushed an update that should fix the time in the container if you specify the timezone in the compose, but if it doesn’t work I’ll have to go back to the drawing board on that one.

On starting the container now, it shouldn’t do anything until the next scheduled run - so it probably won’t do or say anything at all in docker logs netsky until your first cron time. Even with the wrong time zone showing, it should run at scheduled actual time.

I disabled the ‘run on start’ because I couldn’t test the scheduling because it kept running each time I recreated the container.

Let me know if you’d prefer it to run immediately or keep it automatic/scheduled. I might try and introduce a ‘run once’ or ‘run on container start’ env option if I am able.

Thanks so much for testing!

[mgrimace]

quick follow-up ok the docker container date should be correct, but I see the script itself is using a different timezone in my run - I’ll follow-up. In the meantime cron will/should start at the proper local time, so it shouldn’t impact function.

You can check the container date with docker exec -it netsky date. Mine is showing the correct timezone but the script is running and showing the wrong time.

[Ryely-Squires]

The run on container start option would be useful. It'd make troubleshooting a ton easier, as I'm having trouble with getting the script to run. The date is correct on the container.

[mgrimace]

The run on container start option would be useful. It'd make troubleshooting a ton easier, as I'm having trouble with getting the script to run. The date is correct on the container.

Ok I'll do that and try and get it up today (or revert to previous behaviour if I can't make it run properly with the var). In the meantime, just in case when you rebuild a container:

docker stop netsky && docker rm netsky
docker system prune -a -f
docker compose up -d --build

the first two lines should fully stop, remove, and cleanup any leftovers from previous builds, the third line will rebuild using your newest dockerfile. Note the system prune command will remove untagged networks and other images that aren't used/running

[mgrimace]

@Ryely-Squires the latest update (April30) on my test should fix the following:

• The script should again, by default, run on container start, then continue per schedule (tested briefly, working on my end).
• This defaults to true; however, can be optionally over-ridden in the compose.yaml with the env var RUN_ON_START=false. If set to false, the script should only run on the specified cron schedule (untested).
• I also passed the TZ right into the npm start command, rather than just the container overall so it should show the proper time (tested; shows up proper on my end). Haven't tested whether TZ shows properly in the script on a scheduled run - I'll likely have to fix that too, but this should give you the basic functionality now.

If testing, please do the following to remove previous test containers, remove leftover images, and fully rebuild the container:

docker stop netsky && docker rm netsky
docker system prune -a -f
docker compose up -d --build

[Ryely-Squires]

Hey, sorry. Back at the machine.

The timezone is proper on the script when ran manually. But it still gets stuck at "Password entered successfully" while the main branch's script seems to work.

Specifically:

[5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN] Bot started with 1 clusters [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN-WORKER] Started tasks for account XXXXXXXXXX (censored) [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [BROWSER] Created browser with User-Agent: "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0" [5/4/2024, 8:19:55 PM] [PID: 25] [LOG] [MAIN] Starting DESKTOP browser [5/4/2024, 8:20:08 PM] [PID: 25] [LOG] [LOGIN] Email entered successfully [5/4/2024, 8:20:11 PM] [PID: 25] [LOG] [LOGIN] Password entered successfully

Any ideas?

[mgrimace]

Any ideas?

AFAIK this is a 2FA issue.

If you have 2FA enabled, there's no way (at the moment) that I'm aware of to enter the code in headless mode (which isn't advised for docker as it is) - the easiest solution would be to disable 2FA on your account temporarily. As a first step, you could possible run the script manually without docker using headless=false and enter the login code into the browser manually to (hopefully) save the credentials, then rebuild/re-run docker with that fresh config folder. I'm not certain that will work, in fact, give that you can login when it's run manually I'll assume it's not working this way.

See issues #62 and #23 - as it is, the script doesn't have a way (AFAIK) to pass the 2FA code in headless mode at all (so at the moment I don't have a way when it's containerized)

Tl;dr try disabling 2FA on your rewards account and rerunning container