Questions tagged [amazon-s3]
Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.
amazon-s3
50,919
questions
0
votes
0
answers
6
views
Disussion: Data ingestion from sharepoint to Snowflake
Hi looking for suggestions to ingest data from sharepoint to Snowflake.
Currently we have power automate workflow which triggers the data copy from sharepoint to S3. Then we use snowpipe copy command ...
- 155
0
votes
0
answers
9
views
Uploading from OneDrive/Sharepoint in Laravel App
I'm looking for a solution to allow users to authenticate then upload files from their Sharepoint/OneDrive to S3 within a Laravel application. Has anyone been able to accomplish this before?
I've only ...
0
votes
0
answers
8
views
How to connect Athena ODBC using AWS learner lab
I downloaded the Athena ODBC to connect to amazon athena. When I go to configure a data source in ODBC data sources 64bit on my own computer, I can't seem to get the correct secret access key or ...
-3
votes
1
answer
11
views
How can I upload a CSV file from a local folder to a specific folder within an S3 bucket using Python?
If I have a CSV file stored in a specific local folder, how can I upload it to a specific folder within an S3 bucket using Python?
- 177
0
votes
1
answer
16
views
How much data in S3 will be backup for the first time?
Let say I have a S3 bucket which I created it and uploaded files to it a year ago. I kept it unchanged for a year. Now I turn on backup on it using continuous backup and set the total retention period ...
- 215
0
votes
0
answers
17
views
Uploading static files to an S3 as part of a Serverless deploy
I'm extending an existing tool that runs as a bunch of lambdas. It has several components to it (sftp handling, RDS Postgres updates, etc) and a small UI element. The UI needs has a few pages that now ...
- 4,635
0
votes
0
answers
24
views
Unable to read a dataframe from s3
I am getting the following error:
24/07/25 21:29:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/07/25 21:29:53 ...
- 7
-1
votes
1
answer
25
views
Does it make sense to "Let Cloudfront access S3 with VPC endpoint"
Currently I am making an infrastructure with Cloudfront infront of S3 for serving public static assets.
The customer requested that they don't like to have their public static assets being public on ...
- 4,942
0
votes
0
answers
37
views
Unable to upload images to S3 using boto3
I am using fastapi to upload images to my s3 bucket. first i tried presigned url method to upload images from postman and it is working fine. But when i try to directly upload image(size ~7kb) from ...
- 33
1
vote
0
answers
11
views
Python - Writing data from Firehose lambda to S3 with dynamic names
I am using Amazon Data Firehose with a Lambda function to massage data before writing to S3 destination. I need to change the name of the S3 file written to S3 based on the data processed in the ...
0
votes
1
answer
7
views
how to load a ETL script to S3 bucket using yaml CloudFormation stack
I have been writing CloudFormation Stack using yaml and deploying it to AWS Infrastructure ( For legacy reasons, I can not switch to CDK unfortunately ;))
Following yaml code is a part of the ...
- 1,240
-1
votes
1
answer
33
views
pd.to_datetime() not consistently working to convert objects
I have been working with this data (csv) that exists in an AWS S3 bucket. When I am pulling the data I have to transform all the columns to their correct dtypes.
All other dtypes are working properly ...
- 141
0
votes
0
answers
5
views
AWS Cloudfront S3 for Apple App Site Association NoSuchKey
I'm using my React web app (which is hosted using AWS Cloudfront and S3) to host my apple-app-site-association for my iOS app.
I know that I have it stored on S3 correctly because when I go to https://...
- 41
0
votes
0
answers
24
views
Difficulties Encrypting and Decrypting Files from AWS S3 using AWS KMS
[ Explanation ]
I am working on a problem that is proving to be pretty cumbersome, I feel as though I'm getting close to a solution but there's still difficulties I'm encountering.
I currently have a ...
- 17
0
votes
1
answer
26
views
AWS Serverless framework, stack deploy stuck "CREATE_IN_PROGRESS"
I have a problem while using the Serverless framework to deploy my stack to AWS.
I'm running this serverless deploy --stage tst --region eu-west-3 --verbose
Then 2 of my functions get stuck in "...
0
votes
1
answer
21
views
Reading Parquets From S3 With Apache Spark Slows Down At Later Stages
I have millions of parquets files on s3 with directory structure as code/day=xx/hour=/*.parquets.
At max under hour folder we have 2000 parquest file with average size of 100kb.
I am not able to ...
- 1
1
vote
0
answers
26
views
Not able to Access S3 image via Cloudfront
I have created S3 bucket with public access with below mentioned Bucket policy and CORS. My bucket image is accessible in my web app.
But When I create cloudfront distribution for this bucket with ...
- 11
-3
votes
0
answers
13
views
hls.js not resolving segment file paths correctly for files hosted on S3 and accessed via cloudfront [closed]
I've a HLS video upload on an S3 bucket with following folder structure
https://bucket-name.us-west-1.amazonaws.com/folder/path/to/video/output.m3u8
The contents of output.m3u8 file as following:
#...
- 1
0
votes
0
answers
11
views
Amazon Managed Service for Apache Flink keeping S3 as data source
Is it possible to read files from S3 whenever a new file is added from AWS Managed Apache Flink and I need to process the list of new files data one by one, the final result needs to be added to ...
- 495
0
votes
0
answers
16
views
How do I ingest data from singstat into an S3 bucket using AWS Lambda?
this is the code that i have been using:
import json
import urllib.request
import boto3
import os
from datetime import datetime
def lambda_handler(event, context):
# Define the SingStat API
...
0
votes
0
answers
48
views
Spark EOF Error (Parquet Read from S3)- Spark to Pandas conversion
I am reading close to 1 million rows stored in S3 as parquet files into a dataframe (900 MB size data in a bucket). Filtering the dataframe based on values and then later converting to a Pandas ...
- 132
0
votes
0
answers
8
views
How to Efficiently Upload Files in Parallel in Next.js/Node.js Using Async and S3?
I'm working on a Next.js project where I need to upload multiple files to AWS S3 in parallel. I'm using the @aws-sdk/client-s3 for uploading the files. Despite trying different approaches, my uploads ...
0
votes
2
answers
31
views
Too many "Authorized committer" errors after upgrading to Pyspark==3.5.1
The problem
I have recently upgraded my apps to run on Spark3.5.1+YARN3.3.6, and observing frequent failures saying "Authorized committer". The apps run PySpark and I observe the error ...
- 2,202
0
votes
0
answers
16
views
Amazon S3 bucket requirements for create spot-datafeed-subscription
I want to create a spot-datafeed-subscription with the documentation on Track your Spot Instance costs by subscribing to a data feed - Amazon Elastic Compute Cloud.
I am kind of lost in how I have to ...
0
votes
0
answers
11
views
To upload the Google Drive images to AWS S3 using Laravel import functionality
I'm trying to upload the Google Drive images to AWS S3 using Laravel import functionality. I have more than 2000 products in it. Each product has 2 mandatory images, in which I'm facing one issue. The ...
0
votes
1
answer
42
views
S3 Intelligent - Tiering with snowflake managed iceberg tables
Trying to save on huge rarely used data storage costs, with snowflake managed iceberg tables.
Would you apply the economical s3 intelligent-Tiering storage,
to the iceberg’s external volume on s3,
...
- 13
-3
votes
0
answers
23
views
What is the server-side modified date of a file in the AWS S3 cloud? [closed]
Files in the AWS S3 cloud have two different dates/time listed in their properties, the "Server-side modified" and the "Client-side modified". I would like to know what is the ...
- 1
0
votes
0
answers
6
views
I have react-image-crop problems with IOS
It works perfectly in Android and desktop browsers, but I have issues with Iphone and Safari Mac
I am uploading an image cropping with react-image-crop and uploading to S3 AWS
Do you guys know any ...
1
vote
0
answers
32
views
PHP: Does move_uploaded_file() actually work with the S3 Stream Wrapper?
The Amazon S3 stream wrapper "enables you to store and retrieve data from Amazon S3 using built-in PHP functions, such as file_get_contents, fopen, copy, rename, unlink, mkdir, and rmdir. ... ...
- 3,245
1
vote
0
answers
21
views
S3 server simulation in FastAPI, Python
I want to implement S3 server using FastAPI and Python. This is my code for signature validation:
async def verify_signature(request: Request):
try:
authorization_header = request.headers.get(&...
- 21