Questions tagged [azure-data-factory]
Microsoft Azure Data Factory is a cloud service used to invoke (orchestrate) other Azure or external services in a controlled way using the concept of time slices. ADF also provides means to transform data in a parallel way. Data factories are predominately developed using hand crafted JSON, this provides the tool with instructions on what activities to perform.
azure-data-factory
9,421
questions
0
votes
0
answers
18
views
Getting Copying from Column may have data truncation warning during mapping and type Float not compatible error in Data Factory under MS Fabric?
I am trying to create a pipeline in Data Factory service present under Microsoft Fabric in its trail license.
This file is supposed to copy the data present in a file which is in *.txt.gz format ...
- 1,103
0
votes
1
answer
23
views
Getting error at data flow level while extracting data through REST API
I have an ADF pipeline with a REST API call that is supposed to return data in chunks, with a maximum of 100 records per page.
In the below image it is showing until the Set No of pages required ...
0
votes
1
answer
28
views
Azure Data Factory Expression Builder: Error Structuring JSON with Nested Arrays and Objects
I've been working on this for a couple of days now and am reaching out for assistance. I was unable to structure the format properly and need help.
I've been trying to create a JSON structure in Azure ...
0
votes
1
answer
27
views
Azure databricks clusters are failing to load while creating a linked service in azure data factory
This is the error
I'm trying to learn azure data bricks, data factory...
I'm trying to link the data bricks cluster to data factory, but the clusters are failing to load, I've entered the cluster ...
0
votes
1
answer
46
views
Dynamic SQL query in ADF pipeline
I am trying to replace a table in a SQL query dynamically in ADF pipeline. I tried these queries to get it:
select *
from c
where timestampToDateTime("@{concat(variables('lastUpdatedColumn'),'*...
- 535
0
votes
1
answer
32
views
How to supply a certificate while creating a linked service to presto in Azure Datafactory
I am trying to connect to Presto using linked service in Azure Data Factory using the link below:
https://learn.microsoft.com/en-us/azure/data-factory/connector-presto?tabs=data-factory
I have entered ...
- 1,525
0
votes
1
answer
38
views
Rest call failed with client error, status code 404 in ADF. But I can see the date through postman
I am trying to connect share point through graph api using service principal. I have set up a rest dataset in the copy activity and set the base url as https://graph.microsoft.com/v1.0/ and gave the ...
1
vote
3
answers
46
views
Incrementally copy new files by LastModifiedDate with Azure Data Factory
[enter image description here] (https://i.sstatic.net/MBwbT11p.png)
"I need to copy files under the condition that only the files that were last modified are copied. Since I have a total of 6 ...
0
votes
0
answers
22
views
Is Azure Data Factory HubSpot connector limited to 5900 contacts? [closed]
I am using the HubSpot source connector in Azure Data Factory for Companies and Contacts and associated properties.
I have over 7,000 contacts in my HubSpot instance, however when the source connector ...
-1
votes
1
answer
36
views
How to write data to share point list using Azure Data factory
I have a requirment where I need to write data to share point list file using azure data factory , could you please some one help here ?
Is there any way which help to write the data in share point ...
-1
votes
0
answers
37
views
What are the optimization techniques in ADF during copy activity [closed]
we have a scenerio where we have to copy 100 GB of files from one ADLS location to other ADLS location.
Inside the copy activity -> settings I kept DIU and parallelism to maximum level. Is there ...
- 343
0
votes
1
answer
46
views
How to redirect multiple files to multiple folders based on their name using ADF
I wanted to know how to redirect the multiple files into multiple folders in Data Lake using Azure Datafactory.
For example, My file is bag.csv, then I can redirect it into TotalBags kind of folder in ...
0
votes
0
answers
57
views
Why does my Azure Linked Service connection to Hubspot Public App fail?
I have created a public app in Hubspot and connected it to my Hubspot account (essentially followed this (only difference is that I create a public app since I believe this tutorial isn't up to date ...
- 129
0
votes
0
answers
45
views
Error Loading ORC Files into BigQuery: "Invalid timestamp value" after 17th July 2024
I've been successfully fetching ORC files from an Azure Blob Storage to a GCS bucket using the Storage Transfer Service. This process handles around 100 files. After loading the files into GCS, I use ...
- 37
0
votes
1
answer
47
views
Azure Data Factory: Property selection is not supported on values of type 'Integer'
I am trying to implement a load that is split into batches.
I have used the following manual:
https://techcommunity.microsoft.com/t5/azure-synapse-analytics-blog/extracting-sap-data-using-odata-part-4-...
- 145