Race conditions are an entire class of subtle bugs that can and frequently do happen in multithreaded code. Race conditions happen because the programmer has not sufficiently protected
data accesses to prevent threads from interfering with each other. You need to take extra steps
when writing threaded code to ensure things are thread-safe.
What’s going on here is that the operating system is controlling when your thread runs and
when it gets swapped out to let another thread run. This thread swapping can occur at any
point, even while doing sub-steps of a Python statement. As a quick example, look at this
function:
import concurrent.futures
counter = 0
def increment_counter(fake_value):
global counter
for _ in range(100):
counter += 1
if _name_ == "_main_":
fake_data = [x for x in range(5000)]
counter = 0
with concurrent.futures.ThreadPoolExecutor(max_workers=5000) as executor:
executor.map(increment_counter, fake_data)
This code is quite similar to the structure you used in the threading example above. The
difference is that each of the threads is accessing the same global variable counter and
incrementing it. Counter is not protected in any way, so it is not thread-safe.
In order to increment counter, each of the threads needs to read the current value, add one to
it, and the save that value back to the variable. That happens in this line: counter += 1.
Because the operating system knows nothing about your code and can swap threads at any
point in the execution, it’s possible for this swap to happen after a thread has read the value
but before it has had the chance to write it back. If the new code that is running modifies
counter as well, then the first thread has a stale copy of the data and trouble will ensue.
As you can imagine, hitting this exact situation is fairly rare. You can run this program
thousands of times and never see the problem. That’s what makes this type of problem quite
difficult to debug as it can be quite hard to reproduce and can cause random-looking errors to
show up.
As a further example, I want to remind you that requests.Session() is not thread-safe. This
means that there are places where the type of interaction described above could happen if
multiple threads use the same Session. I bring this up not to cast aspersions on requests but
rather to point out that these are difficult problems to resolve.