Questions tagged [amazon-s3]
Amazon S3 (simple storage service) is an online object storage service from Amazon Web Services. QUESTIONS MUST BE ABOUT PROGRAMMING. Questions about general S3 support, functionality, configuration, etc. are OFF-TOPIC.
amazon-s3
50,919
questions
0
votes
0
answers
6
views
Disussion: Data ingestion from sharepoint to Snowflake
Hi looking for suggestions to ingest data from sharepoint to Snowflake.
Currently we have power automate workflow which triggers the data copy from sharepoint to S3. Then we use snowpipe copy command ...
- 155
0
votes
0
answers
9
views
Uploading from OneDrive/Sharepoint in Laravel App
I'm looking for a solution to allow users to authenticate then upload files from their Sharepoint/OneDrive to S3 within a Laravel application. Has anyone been able to accomplish this before?
I've only ...
0
votes
0
answers
8
views
How to connect Athena ODBC using AWS learner lab
I downloaded the Athena ODBC to connect to amazon athena. When I go to configure a data source in ODBC data sources 64bit on my own computer, I can't seem to get the correct secret access key or ...
-3
votes
1
answer
11
views
How can I upload a CSV file from a local folder to a specific folder within an S3 bucket using Python?
If I have a CSV file stored in a specific local folder, how can I upload it to a specific folder within an S3 bucket using Python?
- 177
0
votes
1
answer
16
views
How much data in S3 will be backup for the first time?
Let say I have a S3 bucket which I created it and uploaded files to it a year ago. I kept it unchanged for a year. Now I turn on backup on it using continuous backup and set the total retention period ...
- 215
0
votes
0
answers
18
views
Uploading static files to an S3 as part of a Serverless deploy
I'm extending an existing tool that runs as a bunch of lambdas. It has several components to it (sftp handling, RDS Postgres updates, etc) and a small UI element. The UI needs has a few pages that now ...
- 4,635
0
votes
0
answers
24
views
Unable to read a dataframe from s3
I am getting the following error:
24/07/25 21:29:43 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
24/07/25 21:29:53 ...
- 7
-1
votes
1
answer
25
views
Does it make sense to "Let Cloudfront access S3 with VPC endpoint"
Currently I am making an infrastructure with Cloudfront infront of S3 for serving public static assets.
The customer requested that they don't like to have their public static assets being public on ...
- 4,942
0
votes
0
answers
37
views
Unable to upload images to S3 using boto3
I am using fastapi to upload images to my s3 bucket. first i tried presigned url method to upload images from postman and it is working fine. But when i try to directly upload image(size ~7kb) from ...
- 33
1
vote
0
answers
11
views
Python - Writing data from Firehose lambda to S3 with dynamic names
I am using Amazon Data Firehose with a Lambda function to massage data before writing to S3 destination. I need to change the name of the S3 file written to S3 based on the data processed in the ...
0
votes
1
answer
7
views
how to load a ETL script to S3 bucket using yaml CloudFormation stack
I have been writing CloudFormation Stack using yaml and deploying it to AWS Infrastructure ( For legacy reasons, I can not switch to CDK unfortunately ;))
Following yaml code is a part of the ...
- 1,240
-1
votes
1
answer
33
views
pd.to_datetime() not consistently working to convert objects
I have been working with this data (csv) that exists in an AWS S3 bucket. When I am pulling the data I have to transform all the columns to their correct dtypes.
All other dtypes are working properly ...
- 141
0
votes
0
answers
5
views
AWS Cloudfront S3 for Apple App Site Association NoSuchKey
I'm using my React web app (which is hosted using AWS Cloudfront and S3) to host my apple-app-site-association for my iOS app.
I know that I have it stored on S3 correctly because when I go to https://...
- 41
0
votes
0
answers
24
views
Difficulties Encrypting and Decrypting Files from AWS S3 using AWS KMS
[ Explanation ]
I am working on a problem that is proving to be pretty cumbersome, I feel as though I'm getting close to a solution but there's still difficulties I'm encountering.
I currently have a ...
- 17
0
votes
1
answer
27
views
AWS Serverless framework, stack deploy stuck "CREATE_IN_PROGRESS"
I have a problem while using the Serverless framework to deploy my stack to AWS.
I'm running this serverless deploy --stage tst --region eu-west-3 --verbose
Then 2 of my functions get stuck in "...
0
votes
1
answer
21
views
Reading Parquets From S3 With Apache Spark Slows Down At Later Stages
I have millions of parquets files on s3 with directory structure as code/day=xx/hour=/*.parquets.
At max under hour folder we have 2000 parquest file with average size of 100kb.
I am not able to ...
- 1
1
vote
0
answers
26
views
Not able to Access S3 image via Cloudfront
I have created S3 bucket with public access with below mentioned Bucket policy and CORS. My bucket image is accessible in my web app.
But When I create cloudfront distribution for this bucket with ...
- 11
-3
votes
0
answers
13
views
hls.js not resolving segment file paths correctly for files hosted on S3 and accessed via cloudfront [closed]
I've a HLS video upload on an S3 bucket with following folder structure
https://bucket-name.us-west-1.amazonaws.com/folder/path/to/video/output.m3u8
The contents of output.m3u8 file as following:
#...
- 1
0
votes
0
answers
11
views
Amazon Managed Service for Apache Flink keeping S3 as data source
Is it possible to read files from S3 whenever a new file is added from AWS Managed Apache Flink and I need to process the list of new files data one by one, the final result needs to be added to ...
- 495
0
votes
0
answers
16
views
How do I ingest data from singstat into an S3 bucket using AWS Lambda?
this is the code that i have been using:
import json
import urllib.request
import boto3
import os
from datetime import datetime
def lambda_handler(event, context):
# Define the SingStat API
...
0
votes
0
answers
48
views
Spark EOF Error (Parquet Read from S3)- Spark to Pandas conversion
I am reading close to 1 million rows stored in S3 as parquet files into a dataframe (900 MB size data in a bucket). Filtering the dataframe based on values and then later converting to a Pandas ...
- 132
0
votes
0
answers
8
views
How to Efficiently Upload Files in Parallel in Next.js/Node.js Using Async and S3?
I'm working on a Next.js project where I need to upload multiple files to AWS S3 in parallel. I'm using the @aws-sdk/client-s3 for uploading the files. Despite trying different approaches, my uploads ...
0
votes
2
answers
31
views
Too many "Authorized committer" errors after upgrading to Pyspark==3.5.1
The problem
I have recently upgraded my apps to run on Spark3.5.1+YARN3.3.6, and observing frequent failures saying "Authorized committer". The apps run PySpark and I observe the error ...
- 2,202
0
votes
0
answers
16
views
Amazon S3 bucket requirements for create spot-datafeed-subscription
I want to create a spot-datafeed-subscription with the documentation on Track your Spot Instance costs by subscribing to a data feed - Amazon Elastic Compute Cloud.
I am kind of lost in how I have to ...
0
votes
0
answers
11
views
To upload the Google Drive images to AWS S3 using Laravel import functionality
I'm trying to upload the Google Drive images to AWS S3 using Laravel import functionality. I have more than 2000 products in it. Each product has 2 mandatory images, in which I'm facing one issue. The ...
0
votes
1
answer
42
views
S3 Intelligent - Tiering with snowflake managed iceberg tables
Trying to save on huge rarely used data storage costs, with snowflake managed iceberg tables.
Would you apply the economical s3 intelligent-Tiering storage,
to the iceberg’s external volume on s3,
...
- 13
-3
votes
0
answers
23
views
What is the server-side modified date of a file in the AWS S3 cloud? [closed]
Files in the AWS S3 cloud have two different dates/time listed in their properties, the "Server-side modified" and the "Client-side modified". I would like to know what is the ...
- 1
0
votes
0
answers
6
views
I have react-image-crop problems with IOS
It works perfectly in Android and desktop browsers, but I have issues with Iphone and Safari Mac
I am uploading an image cropping with react-image-crop and uploading to S3 AWS
Do you guys know any ...
1
vote
0
answers
32
views
PHP: Does move_uploaded_file() actually work with the S3 Stream Wrapper?
The Amazon S3 stream wrapper "enables you to store and retrieve data from Amazon S3 using built-in PHP functions, such as file_get_contents, fopen, copy, rename, unlink, mkdir, and rmdir. ... ...
- 3,245
1
vote
0
answers
21
views
S3 server simulation in FastAPI, Python
I want to implement S3 server using FastAPI and Python. This is my code for signature validation:
async def verify_signature(request: Request):
try:
authorization_header = request.headers.get(&...
- 21
0
votes
0
answers
27
views
Data Integrity S3 During Transit for Large File
I'm new to encryption, so my question might seem basic. Despite reading multiple articles, I'm still unsure if I fully understand.
I have an S3 bucket, bucket_a, which will be used by different AWS ...
- 511
-1
votes
0
answers
35
views
+50
Uploading the build artifacts to S3 is failing in sam deploy
I am using sam for setting up infrastructure. During sam deploy, uploading the artifacts to s3 bucket is failing with error
botocore.exceptions.ClientError: An error occurred (403) when calling the ...
- 118
0
votes
0
answers
18
views
AWS Amplify Gen 2, NextJS: How to Upload images from the NextJS backend API Route?
I'm using Amplify Gen 2 for my NextJS project.
I have been searching through all the docs but didn't find any example on how to upload files that comes to the NextJS backend API route. I tried the &...
- 927
-3
votes
0
answers
18
views
Upload problem in react project from china [closed]
I have a react portal , file upload successfully from india and us before few days file upload from china but few days he did not upload file on our react portal what's the issue, Issue same from s3 ...
0
votes
0
answers
26
views
Error uploading file to s3 storage using minio
I am trying to upload two files. Both have the same extension, but one is much smaller than the other. One is just 100 kb, and the other is 100 MB. The extension is .pb, and these files are part of a ...
- 368
0
votes
0
answers
17
views
SignaturDoesNotMatch when trying Multipart upload in AWS S3 bucket in flutter
The problem you're facing is related to AWS S3 multipart upload signature calculation. Specifically, the error SignatureDoesNotMatch indicates that the AWS signature you're generating does not match ...
-1
votes
1
answer
22
views
com.amazonaws.services.s3.model.AmazonS3Exception: SignatureDoesNotMatch Error When Uploading Files to S3 from Android App
I'm encountering a SignatureDoesNotMatch error when trying to upload a file to Amazon S3 using the AWS SDK for Java as I'm developing an android app with a feature that allows users to upload their ...
0
votes
0
answers
15
views
crawler successfully executes even after error
I am facing a weird issue,
i forgot to add [kms:*] in my AWS crawler policy and because of that there was errors,
but even after that crawler succeeded , without creating any table :
this is the ...
- 851
0
votes
0
answers
15
views
How to Get File Type passed into multipart form-data with Multer When Uploading to S3 in Node.js?
I'm using Multer to handle file uploads in my Node.js application, and I'm uploading files to an S3 bucket. I need to get the type I pass into multi-part formdata before uploading. How can I achieve ...
-1
votes
1
answer
18
views
How to Share Specific Documents from a Private S3 Bucket as Attachments in Chat?
I'm working on an application where I upload images and documents to an Amazon S3 bucket that is set to private for security reasons. I need to securely share some specific documents as attachments in ...
0
votes
0
answers
65
views
Failed to load module script: Expected a JavaScript module script but the server responded with a MIME type of "text/html" react vite
I am deploying my react app in AWS S3 bucket.
Everything works fine locally but each time a new build is deployed, then i get the error as :
Failed to load module script: Expected a JavaScript module ...
0
votes
1
answer
66
views
Received response status [FAILED] from custom resource. Message returned: Command died with <Signals.SIGKILL: 9>
What am I trying to do
I am using CDK to build a stack that can run a python app
EC2 to run the python application
RDS instance to run the PosgreSQL database that connects with EC2
Custom VPC to ...
- 5,943
0
votes
0
answers
15
views
Upload Pdf file to Amazon S3 using Nextjs route handler
I've been attempting to upload a PDF to S3 using a Next.js route handler. However, I'm encountering a server error 500 when sending the post request. When I tried sending the request without the PDF ...
0
votes
0
answers
23
views
How do you get the object metadata for an S3 Bucket object when querying from javascript
I am getting an object from an S3 bucket using the GetObject command. I want to query some user defined metadata on the object.
I can see that the metadata is included in the response headers but I ...
- 21
-1
votes
0
answers
23
views
why is s3 returning null even though I have folders in the directory
I made this small code where the code finds the folder i write in the input box and displays it's content but aws is just sending no directory data to me
Here is the function doing the work
async ...
1
vote
1
answer
37
views
Presigned URL PutObject with Flutter and Golang
I'm trying to upload images to my S3 bucket using presigned URLs. My bucket has no public access, and the CORS policy is this:
[
{
"AllowedHeaders": ["*"],
&...
- 226
0
votes
2
answers
40
views
CopyObjectCommand does not copy the file Returns Error Header not implemented
In my Nextjs project I'm trying to copy files using (CopyObjectCommand).
Using @aws-sdk/client-s3: 3.616.0
The PutObjectCommand and DeleteObjectCommand are working fine. However, the CopyObjectCommand ...
0
votes
1
answer
16
views
AWS api 403 Forbidden in S3, Cloudfront
I've deployed a React application on S3 and a Spring Boot API on EC2, using CloudFront as a CDN. While I can access the static content (HTML, CSS) through CloudFront, I'm unable to fetch API data. ...
- 29
0
votes
0
answers
14
views
can gdal.Driver handle S3 file paths?
A project that I am working on requires me to access geometries stored in a .gdb folder with OGR/GDAL as geopandas is currently not capable of accessing M values associated with those geometries.
I've ...
0
votes
1
answer
27
views
Multer s3 upload: File in s3 not playing and file increase
I am uploading a mp4 file to s3 via node.js s3.
The code works and I see the file in the s3. However The original file is 860kb but when uploading it it turns to 1.4MB and when I download it from S3 ...
- 3,366