Questions tagged [concurrent.futures]
concurrent.futures is a Python module which provides a high-level interface for asynchronously executing callables.
concurrent.futures
986
questions
0
votes
0
answers
10
views
Python SSL Verify = False Not Working w/ Trino
I'm encountering a super weird issue, where while using trino, the verify=False argument doesn't work when running in a separate python thread / process. When running the script directly, it works ...
- 33
0
votes
1
answer
34
views
Recursive use of ProcessPoolExecutor is Hanging
Problem
I'm trying to use a ProcessPoolExecutor with recursive calls, but it doesn't work. I created a minimal example below
from concurrent.futures import ProcessPoolExecutor
from time import sleep
...
- 159
0
votes
0
answers
29
views
Calling python "concurrent.futures" multiprocessing from C# opens multiple winform instances of the application
pythonnet by Python.Runtime 3.0.3 (Installed the winform application in Visual Studio 2022)
adfuller_test.py (This is my python script file)
I am calling python code from a winform application which ...
- 1,269
0
votes
1
answer
56
views
Getting the result from a future in Python
I have the following code which executes a process and calls a callback function when the process is done
import os
import subprocess
import tempfile
def callback(future):
print(future....
- 1,196
0
votes
0
answers
28
views
When using the Python Concurrent Futures should I use ALL the vCores?
Is it safe when using the Python Concurrent Futures to use ALL the vCores available in the machine?
My code currently uses the vCores available minus 1 as I want to leave one vCore "free" ...
- 678
0
votes
1
answer
55
views
Python Concurrent Futures not taking the data out of memory
I am working on Python code that copies tables of 50+ GB from PostgreSQL to SQL server and creates tables dynamically. Copies the data using SQLAlchemy Streaming and batch insert using Concurrent ...
0
votes
0
answers
11
views
Can processing of request/grequest responses be run concurrently in Python2?
I'm writing a Python2 program that pulls a large amount of JSON data from a remote CouchDB database. The data is indexed by timestamp: I pass a startkey and an endkey, and the database returns all ...
- 303
0
votes
0
answers
27
views
Passing Spark Session variable as a parameter while executing a function using ProcessPoolExecutor is not working
Pyspark version: 3.3.0-amzn-0
Python: 3.7.16
I am using the below code snippet where I am trying to use spark session as a parameter while calling a function by name test using ProcessPoolExecutor.
...
- 99
0
votes
0
answers
22
views
Improve Parallelising *Reading, Cropping and Patching* individual .RT-H5 files (using Python **H5py**)
I have a data processing pipeline that is a perfect candidate for parallelisation but I can't seem to get "good" speed ups.
The process that I need to complete is:
Read in a .RT-H5 file ...
- 131
0
votes
0
answers
20
views
How can I transform a petl table across all CPUs?
I created an ETL pipeline using Python ETL PETL.
It works as expected, but my current transformation part is a bit slow.
I'm utilizing ProcessPoolExecutor to transform each petl data table in separate ...
- 246
1
vote
1
answer
57
views
Can I create a global ThreadPoolExecutor in a Flask application?
I am using a concurrent.futures.ThreadPoolExecutor in a WSGI (Flask) Python REST service to send queries in parallel. I use the code below to instantiate one executor per request. There is one ...
- 609
1
vote
0
answers
30
views
How to capture printed messages in parallel process?
I have a function parallel_run which print out the diagnostic message during the run and then return a result. I want to capture the printed message and also the returned value. However, somehow I got ...
0
votes
0
answers
12
views
Skipping any unreachable network device using concurrent.futures.ThreadPoolExecutor in python
With my portion of the code below using concurrent.futures.ThreadPoolExecutor
if __name__ == "__main__":
xw.Book("Main_Template.xlsm").set_mock_caller()
test1 = ...
0
votes
0
answers
113
views
Finding the valid tetrahedrons with smallest volume for the large input file size
I have two files points_small.txt and points_large.txt which contain the list of points on a 3D plane. Each point is defined by its coordinates and an associated number and is presented in the ...
- 139
0
votes
1
answer
57
views
Multithreading stuck at last future
def download_files_from_folder(base_url, folder_name):
folder_url = f"{base_url}{folder_name}/"
response = requests.get(folder_url)
soup = BeautifulSoup(response.content, "...
- 11