python [Bank Account] Threading Tests Incorrectly Passing on Website, Correctly Failing Locally

The bank account problem is meant to be "hard" with the difficulty coming from the need to lock the account during threading.

I created a solution that passed all of the other tests and completely ignored the threading part (there was no code at all for that). On my local PC it passed all tests except the threading one, exactly as expected.

Except when I copy/pasted it to exercism and ran the tests there, all of the tests passed and I could submit my solution.

(Edit: Also the Intro blurb for Bank Account says the solution needs to handle "threading/processes" depending on what the language is. Given Python supports both of these but the code is only using Threading, this should probably be changed to just threading).

Dec 19 '21 12:12 mohmad-null

🤖 🤖

Hi! 👋🏽 👋 Welcome to the Exercism Python Repo!

Thank you for opening an issue! 🐍 🌈 ✨

If you are requesting support, we will be along shortly to help. (generally within 72 hours, often more quickly).
Found a problem with tests, exercises or something else?? 🎉
◦ We'll take a look as soon as we can & identify what work is needed to fix it. (generally within 72 hours).

◦ If you'd also like to make a PR to fix the issue, please have a quick look at the Pull Requests doc.
We 💙 PRs that follow our Exercism & Track contributing guidelines!

Here because of an obvious (and small set of) spelling, grammar, or punctuation issues with one exercise,
concept, or Python document?? 🌟 Please feel free to submit a PR, linking to this issue. 🎉

‼️ Please Do Not ‼️

       ❗ Run checks on the whole repo & submit a bunch of PRs.
              This creates longer review cycles & exhausts reviewers energy & time.
              It may also conflict with ongoing changes from other contributors.
       ❗ Insert only blank lines, make a closing bracket drop to the next line, change a word
              to a synonym without obvious reason, or add trailing space that's not an EOL for the very end of text files.         ❗ Introduce arbitrary changes "just to change things" .

...These sorts of things are not considered helpful, and will likely be closed by reviewers.

For anything complicated or ambiguous, let's discuss things -- we will likely welcome a PR from you.
Here to suggest a feature or new exercise?? Hooray! Please keep in mind Chesterton's Fence.
Thoughtful suggestions will likely result faster & more enthusiastic responses from maintainers.

💛 💙 While you are here... If you decide to help out with other open issues, you have our gratitude 🙌 🙌🏽.
Anything tagged with [help wanted] and without [Claimed] is up for grabs.
Comment on the issue and we will reserve it for you. 🌈 ✨

Dec 19 '21 12:12 github-actions[bot]

Hi @mohmad-null,

Thanks for logging this issue!
The TL;DR is that this works "as expected" on a local environment, but does not behave "as expected" within Docker (which is how our test runner works in production). Current working suspicion is that Docker is limiting the number of threads, thereby short circuiting the behavior needed to create a locking issue via the tests, with the side effect that the tests always succeed when the solution is tested via run-in-docker.

This one is tricky, and I have been thinking of soliciting help from folx with more experience in threading/processing within Docker to rework it. Here are some details from a Slack discussion around this from a couple of months ago: https://exercism-team.slack.com/archives/CC5DGSAG6/p1633825675090000. Screengrab below, if you cannot access Slack:

2021-12-19_13-10-46

Happy to hear your thoughts on how to go about fixing this one.

As for the directions -- those are pulled from problem-specifications, so changes would need to be proposed there, or we would need to work out Python-specific language for an addendum that pointed students toward using Threading.

Dec 19 '21 21:12 BethanyG

I had a friend complete the Bank Account one, and like mine, they made no attempt to do the locking thing. Their solution was broadly similar to mine.

For them, all tests passed in their local environment(!), but the concurrency test correctly failed when the code was copy/pasted to exercism. So the exact opposite behaviour to what I was getting.

Dec 21 '21 21:12 mohmad-null

We found similar issues when we were testing/investigating. I could get a pass or a fail on my Mac by reducing thread count as noted in the Slack thread -- but only under some other OS load. Pass/Fail wasn't exactly reliable. @J08K had more reliable luck using Windows. The website seemed more reliably bad/incorrect -- although 1 out of 4 would fail "as expected", with the other three passing.

So definitely a case for someone enthusiastic about multithreading in Python to dig into. 😄 This exercise might need a bit of a re-think testing-wise. LMK if you'd like to take it on. 😉

Dec 22 '21 20:12 BethanyG

@IsaacG @mohmad-null -

I am going to close this. We recently merged canonical data for this exercise in problem-specifications, and as part of the maintenance work, we created a JinJa2 template, a new example, and a tests.toml file. Rather than fiddle around to see if I could create an effective concurrency test against the now much faster Python 3.11, I instead commented out the concurrency test case.

If either of you would like to take on making an effective case, LMK and I will re-open.

Apr 15 '23 02:04 BethanyG