Parallel Processing in Python Scripts
At times it's possible to complete a batch job using parallel processing using fork instead of threads which needs much more careful planning. On the other hand fork is easy to implement, although it might not be as efficient and flexible as threads.
Forking is an important part of *nix design, by forking shells enable us to chain many commands using pipes. The basic idea of forking is to create a clone of the current process, and the newly created processes are called child processes and they can tell themselves apart by checking the return value of the fork() call.
In this article we'll see how to implement forking in Python scripts.
Forking in started with fork() system call, this will create a clone of the running process, the sample code below will help in understanding better.
After fully understanding the basic example, you can experiment with your ideas of parallel processing, I have tried an example where the parent process forks child processes to fetch urls from an array.
You can do much bigger things, your imagination is the limit. Enjoy coding in Python.
|All times are GMT +5.5. The time now is 18:24.|