Need Recommendations for Fine-Tuning LLMs on Azure ML: Best Practices

I need recommendations for each of the following questions:

1. Choice of Models

   • Should I use Hugging Face models or models available in the cloud via an API (e.g., models from Replicate or Azure ML)?

2. Creation of Pipelines for Fine-Tuning

   • How do I create pipelines for LLM models?

   • Is it necessary to create pipelines for fine-tuning a model? Is it similar to creating pipelines for a traditional ML model?

   • How can I do this on Azure ML? Which module allows pipeline creation on Azure ML?

   • Is it possible to do this with open-source tools like Prefect or Apache Airflow?

3. Development Environment

   • Where should I develop my code: in Azure ML or in VS Code? Does it make a difference?

   • If my project structure is as follows:

     Project :
     ├── Notebooks
     │   └── Notebook.ipynb
     ├── Scripts
     │   ├── retreive_data.py  # Retrieve data from a database
     │   ├── preprocess_data.py
     │   ├── train_model.py
     │   ├── evaluate_model.py
     │   ├── schedule_retraining_model.py   # e.g., retrain the model monthly
     │   ├── recover_best_model.py
     │   └── deploy_model.py
     

4. Integration and Deployment

   • Should I commit my code from VS Code to GitHub / Azure DevOps, then retrieve it in Azure ML and deploy it from there?

   • Should I code in Azure ML, create all folders and files, then click the "open in VS Code" button (in the Notebook section of Azure ML)? Then, code in VS Code, save, and the scripts will be added in Azure ML?

   • Should I commit the code from VS Code to GitHub / Azure DevOps, or should I commit from Azure ML?

   • When should I commit the code to GitHub / Azure DevOps, and when is it sufficient to code in Azure ML without committing?

   • Will I need to implement CI/CD? If yes, at which stage and in what context? What is the benefit?

5. Model Deployment

   • To deploy the model and obtain an endpoint for my clients, can I do this directly from Azure ML, or do I need to commit to GitHub / Azure DevOps and then deploy the model?

6. Resource Usage

   • If I use VS Code, which resources should we use? Local resources or those offered in Azure ML?

   • Is it necessary to set up a devcontainer if I use VS Code?

7. Using an API

   • If I commit to GitHub / DevOps, should I add an API with Flask or FastAPI for model inference?

   • In this case, do I need to deploy the code in Azure Web App with CI/CD?

8. Data Storage

   • Where should I store data on Azure? My data consists of PDF documents and their JSON equivalents. Where should I store them?

9. Azure ML Version

   • Which version is recommended for development on Azure ML: Python SDK v1 or v2?