All Questions
4,965
questions
0
votes
0
answers
38
views
Unable to upload images to S3 using boto3
I am using fastapi to upload images to my s3 bucket. first i tried presigned url method to upload images from postman and it is working fine. But when i try to directly upload image(size ~7kb) from ...
- 33
-1
votes
1
answer
33
views
pd.to_datetime() not consistently working to convert objects
I have been working with this data (csv) that exists in an AWS S3 bucket. When I am pulling the data I have to transform all the columns to their correct dtypes.
All other dtypes are working properly ...
- 141
0
votes
0
answers
16
views
How do I ingest data from singstat into an S3 bucket using AWS Lambda?
this is the code that i have been using:
import json
import urllib.request
import boto3
import os
from datetime import datetime
def lambda_handler(event, context):
# Define the SingStat API
...
0
votes
0
answers
48
views
Spark EOF Error (Parquet Read from S3)- Spark to Pandas conversion
I am reading close to 1 million rows stored in S3 as parquet files into a dataframe (900 MB size data in a bucket). Filtering the dataframe based on values and then later converting to a Pandas ...
- 132
1
vote
0
answers
21
views
S3 server simulation in FastAPI, Python
I want to implement S3 server using FastAPI and Python. This is my code for signature validation:
async def verify_signature(request: Request):
try:
authorization_header = request.headers.get(&...
- 21
0
votes
0
answers
26
views
Error uploading file to s3 storage using minio
I am trying to upload two files. Both have the same extension, but one is much smaller than the other. One is just 100 kb, and the other is 100 MB. The extension is .pb, and these files are part of a ...
- 368
0
votes
0
answers
14
views
can gdal.Driver handle S3 file paths?
A project that I am working on requires me to access geometries stored in a .gdb folder with OGR/GDAL as geopandas is currently not capable of accessing M values associated with those geometries.
I've ...
-2
votes
0
answers
25
views
Python downloads empty files from S3 bucket after checking if it already exists
I have a python script running in Windows that downloads files from an AWS S3 bucket to a Windows File Explorer folder. Before the download, I am trying to check if the file is already there. The ...
- 29
1
vote
0
answers
44
views
List all objects in an S3 bucket on MinIO
I have a bucket with about 5'000'000 objects inside (as shown from the MinIO web interface).
I need to traverse every single object (and later on do some processing on them).
To test the functionality,...
- 159
3
votes
1
answer
87
views
How to use fake s3 bucket
I tried to use aiobotocore for async work with Ceph, but i have no access to real bucket, then i need to mock s3 bucket to create a fake one
I'm trying to use moto but it anyway want to connect a real ...
- 31
-1
votes
1
answer
39
views
How do i get my Python .env credentials file into Ubuntu EC2?
I have a .env file which my app.py file references, containing my AWS credentials. How do i get this into my AWS Ubuntu EC2 machine?
I can't even upload it to GitHub.
If there's any security issues ...
- 403
-1
votes
0
answers
24
views
Django hosted on Render - Media Files/template vars
I've got a Django app that stores its static and media files in an AWS S3 bucket. Retrieval, usage and storage of these files work fine locally. The issue is, when accessing through the live site ...
- 35
1
vote
0
answers
45
views
How can invoke a Lambda function with AWS Signature auth without using Postman?
I have a simple Lambda function in Python that converts a text string into speech and saves it to an S3 bucket. I want to then get that mp3 file (or URL) as a response.
I will invoke the Lambda ...
- 19
1
vote
0
answers
21
views
Get File uploaded to AWS S3 bucket and read into Pillow to convert it
I have a Lambda Function which triggers when a file is uploaded to it's bucket. I want to
open said file with Pillow and convert it tp JPEG.
Code looks like this:
import os
import json
import boto3
...
- 105
1
vote
1
answer
86
views
Getting "cannot schedule new futures after interpreter shutdown" error while using boto3.client download_file and upload_file functions
I am using boto3.client.download_file() and boto3.client.upload_file() functions, For python3.8 environment it's working fine, but after upgrading to python3.10, getting "RuntimeError: cannot ...
- 37