Questions tagged [multiprocessing]
Multiprocessing is the use of two or more central processing units (CPUs) within a single computer system. Relevant implementation and usage details vary per operating system and programming language. So always add tags for both the OS and language when using this tag.
multiprocessing
14,733
questions
0
votes
0
answers
25
views
What is the most efficient way to multiprocess over a very large dataframe?
I have a large Dataframe that I need to do a lot of matching operations over, and in the past have always used the below method for doing it. However, the Dataframe that I am currently attempting to ...
- 23
-2
votes
1
answer
20
views
time.perf_counter() or time.perf_counter_ns() giving weird and incorrect values
enter image description here
I have written this simple while I was working with multi-processing.
Well, everything in my code was going perfectly, except the time.perf_counter() giving like 7000 ...
0
votes
0
answers
21
views
Does it make sense to use Python's multiprocessing module when deploying code on Azure?
Our team has deployed a Python script on Azure Machine Learning (AML) to process files stored on an Azure storage account.
Our pipeline consists of a ForEach activity, that calls the Python script for ...
- 4,519
0
votes
0
answers
36
views
Correct use of boost notify_all in interprocess communication on linux kernels [duplicate]
tl;dr should boost notify_all() be used before or after releasing the lock?
Wikipedia has this comment on signal (in my view similar to broadcast):
it is often also acceptable to release the lock ...
0
votes
1
answer
82
views
Fork in Linux: Weird infinite loop while freeing condition variable
Context
System: Fedora 40
Library Used: pthread.h
Details: Read "To the point" if not interested
I apologize for providing no code here as the whole program is more than 4000 lines of code ...
0
votes
1
answer
39
views
Executing a binary semaphore or mutex for context switching operation in python multiprocessing
I'm trying to automate a sync relation between a win application and a java application.
My criteria is :
start win and jav application
execute command in jav application
wait for response from jav ...
1
vote
1
answer
31
views
Using multiprocessing Pool with a function that takes a list of arrays
I am trying to write a function to read a large number of files in parallel. The code I have for this goes like:
import numpy as np
from multiprocessing import Pool
from functools import partial
def ...
- 55
0
votes
1
answer
26
views
Python multiprocessing throwing up unexplained errors
I have a python program which is separated into three files, dielectric_functions.py, routines.py and cartmoments.py, organized as follows --
dielectric_functions.py is the entry point to the program ...
- 11
0
votes
0
answers
17
views
Parallel predictions on the same Keras Model
I have a .keras file (which I will use to load my model) and a huge dataset X (~5m rows) which I want to call model.predict(X) on to retrieve my predictions.
Is there any way to utilise ...
- 1,331
0
votes
1
answer
24
views
R: running a mirai_cluster via parallelly::makeClusterPSOCK
my goal is to perform some heavy computation in R on a cluster of several Linux hosts, using docker containers.
Within R I want to use foreach.
To do this, I believe the best approach is with the ...
- 495
1
vote
1
answer
35
views
Popening a gnome-terminal in python immediately appears as a zombie
For background I am working on a script to train multiple pytorch models. I have a training script that I want to be able to run as a sub process in a gnome terminal. The main reason for this is so I ...
- 445
0
votes
0
answers
32
views
Python multiprocessing is slow on slurm
I'm using Python multiprocessing and then submitting the job with sbatch. The .sh code is as follows:
#!/bin/bash
##----------------------- Start job description -----------------------
#SBATCH --...
0
votes
1
answer
25
views
Joblib parallel processing introduces {built-in method time.sleep} in profiling
I am using Joblib to run parallel jobs in my Python application. During profiling, I noticed that the slowest process was {built-in method time.sleep}. Interestingly, this issue disappears when I ...
- 17.2k
0
votes
1
answer
35
views
How can I manage to run a single processor project with all processor cores?
import multiprocessing, time
mpCores=multiprocessing.cpu_count()
print("This PC has", mpCores,"cores. Currently only utilising 1, upgrades to follow")
tic = time.perf_counter() ...
- 1
0
votes
2
answers
59
views
Python Multiprocessing Self-updating Object
I'm trying to write a display for cardinal directions. I've got an interface that runs at a high refresh rate, and a hardware device that only updates a few times a second.
I need to be able to get ...
- 3