Getting Started.md

ADF Universal Framework

Welcome to ADF Universal Framework on GitHub! Whether you're a seasoned developer or just getting started, this guide will help you quickly get up and running with our project.

Project Description

This project is a universal Azure Data Factory framework designed to simplify and standardize the development and maintenance of ETL tasks. It provides a set of common data processing pipelines and templates that can be applied to various data integration and transformation scenarios.

Quick Start

Prerequisites

Make sure you have the following tools installed and configured: Prerequisites:

• Git

• Pycharm or Visual Studio Code

• Azure Account

• Azure DevOps Account (Repos in Azure DevOps is not used in this case)

• GitHub Account (Config handshake between GitHub and Azure DevOps when create a new pipline in Azure DevOps )

• Install and config Self Hosted Agent - Azure DevOps (For cost saving purpose)
  Reference 1-5
  Once config ready you should see the result as below:
  
  

  Finally, you should see result as below
  

  Note:
  If you are using VPN locally please remove/set git proxy correctly.

  git config --global --unset http.proxy
  git config --global --unset https.proxy
  
  git config --global http.proxy http://your.new.proxy:port
  git config --global https.proxy http://your.new.proxy:port
  
  git config --get http.proxy
  git config --get https.proxy
  

• Install and config ADF Self-Hosted IR locally for cost saving purpose

  1. Go to Integration Runtime
  2. Create New Self-Hosted IR
  3. config by express or manually
     • Download agent installation file or using IR_ADF_ams.exe
     • Express installation as below

Get Started with ADF Universal Framework

 1. xxx
 2. xxx
 3. xxx
 4. xxx
 5. xxx
 6. xxx

Get Started With Metadata DB SQL Project

Get Started With Universal Task Project

Get Started With CI/CD Project