Marketplace Profile

Kaustubh S

Azure Data Bricks Engineer

Kaustubh S is an experienced Azure Databricks Engineer, recognized for his advanced skills in managing and optimizing big data workloads within cloud environments. His expertise in building scalable data pipelines, performing in-depth troubleshooting, and enhancing data processing efficiency has been crucial in driving data-driven insights and operational excellence. Kaustubh is available hourly, monthly, or quarterly to help you streamline your Azure Databricks infrastructure and address any complex data engineering challenges.

AzureLevel 2

Technical Profile

Skills and Delivery Strengths

Review this resource's technical coverage, delivery responsibilities, and fit for partner-led execution.

Technical Skills Certifications Responsibilities

Capability Stack

Technical Skills

18 Skills

AzureAzure data factoryAzure Synapse AnalyticsAzure data bricksAzure Data Factory (Data flow)PYSPARKSQLOracle 11gSQLAzure SQLAzure data lakeDatabricks Delta lakedatabricks LakehouseAzure BLOBdelta table.WindowsAzure DevOpsGitHub

Verified Credentials

Certifications

2 Credentials

Certified Microsoft Azure Data Engineer Associate (DP-203)

Databricks Certified Data Engineer Associate

Delivery Scope

Responsibilities

Currently working as an Azure data engineer, I developed pipelines and codes using Azure data factory and Synapse to ingest data from different sources and transform the data using Databricks according to the end user's needs.

Have Experience creating Linked services to fetch the data from different sources like SQL DB and oracle Databases.

Have enough experience in fetching data from different APIs and loading the data to Azure data lake storage.

Designed and developed Delta live pipelines (ETL) in Databricks to ingest, store, and process data from multiple sources based on Medallion architecture (Bronze, silver, gold).

Perform analyses on data quality and apply business rules in all layers of data extraction transformation and loading process.

Created DevOps pipeline to deploy the codes to higher environments using CI/CD process and proficient in code deployments in development, testing and production environments using Azure DevOps.

Experienced working with Azure BLOB and ADLS container, Data Lake storage and Azure delta tables in Databricks to load data into and manage the folder structure in containers.

Experienced in converting the SQL code from a source system into dataflow transformations in Azure data factory for building pipelines. Used Dataflow transformation in ADF to build the data movement pipeline in ADF.

Hire This Resource

Browse More Talent