Skip to content

Pallavi Goparaju

Senior Databricks Data Engineer • Redmond, WA • linkedin.com/••••• • drivetube.ai/•••••

Professional Summary

Senior Databricks Data Engineer with 5+ years of experience designing, building, and optimizing Databricks and PySpark ETL/ELT pipelines in Azure lakehouse environments. Skilled in Delta Lake, ADLS Gen2, ADF orchestration, dbt refactoring, performance tuning, monitoring and CI/CD to improve pipeline reliability and reporting SLAs.

Technical Skills

Programming Language: Python,SQL
Databases: Snowflake,Azure Synapse Analytics,Azure SQL Database,Partitioning,Query Optimization
Cloud Platforms: Microsoft Azure,AWS,ADLS Gen2,Azure Blob Storage,AWS Glue
DevOps & Infrastructure: Azure DevOps,Azure Key Vault,Azure Monitor,Git
Testing & QA: Record-count checks,Lineage
Data Engineering & Processing: Azure Databricks,Spark SQL,PySpark,Delta Lake,Apache Airflow
Data Analysis & Visualization: Kusto Query Language,Cluster tuning,Notebook workflows,Power BI,DAX
Data Integration & ETL: Data Validation,dbt,Schema Evolution,ADF,Fivetran
Identity & Access Management (IAM): RBAC

Work Experience

Cigna
Connecticut, CT
Data Engineer - Operations Analytics
May 2025 – Present
Worked at a healthcare insurer supporting enterprise analytics and operations reporting by building and maintaining Azure lakehouse ETL/ELT pipelines.
Tech Stack: Azure Data Factory, Azure Databricks, PySpark, dbt, Azure Monitor, KQL, Azure DevOps, Python, SQL
  • Built and supported production ETL/ELT pipelines using Azure Data Factory, Azure Databricks, PySpark, dbt, Python and SQL to ingest and transform enterprise analytics data from multiple source systems for operations analytics.
  • Optimized high-volume PySpark transformations and Databricks cluster configurations to improve nightly processing scalability and reduce downstream reporting bottlenecks.
  • Refactored dbt models and SQL logic for critical reporting workloads, improving query performance by 30% while preserving business metric definitions across analytics layers.
  • Implemented end-to-end validation controls including freshness checks, record-count gates and structured error handling across ADF and Databricks jobs to detect feed issues earlier in the pipeline lifecycle.
  • Used Databricks job logs, Azure Monitor, KQL and SQL reconciliation checks to investigate pipeline failures, isolate anomalous source records and reduce repeated production support escalations.
  • Established CI/CD patterns with Git and Azure DevOps for Databricks notebooks and ADF artifacts to improve release consistency and reduce deployment-related incidents.
Cognizant
Redmond, WA
Senior Data Engineer
Feb 2024 – May 2025
Worked at an IT services provider building data pipelines to move healthcare member and claims data into ADLS Gen2 and curated lakehouse layers for enterprise reporting.
Tech Stack: Azure Databricks, Azure Data Factory, ADLS Gen2, Delta Lake, PySpark, Azure SQL, Power BI, Azure Monitor, KQL
  • Built and maintained Databricks and ADF pipelines to ingest healthcare member and claims data into ADLS Gen2 raw, staged and curated zones for enterprise reporting consumers.
  • Profiled slower nightly workloads and migrated complex transformations into PySpark and Delta Lake, improving production reliability and reducing early-morning SLA overruns.
  • Defined ADLS folder standards, partitioning patterns and curated storage layer conventions to enable predictable automated consumption by Power BI and SQL-based reporting.
  • Troubleshot intermittent ADF and Databricks failures using Azure Monitor, KQL, Databricks job logs and SQL reconciliation checks; identified malformed payloads and integration runtime issues.
  • Optimized SQL used by downstream reports and refactored heavier logic into Azure SQL stored procedures to improve Power BI refresh predictability during business hours.
  • Partnered with BI and compliance stakeholders to define ADLS/Synapse RBAC, encryption defaults and catalog lineage practices for secure access and traceability.
University of Bridgeport
Bridgeport, CT
Graduate Teaching Assistant - Data Engineering Projects
Aug 2022 – Dec 2023
Supported university data engineering projects consolidating clinical, operational and research datasets for healthcare analytics and research reporting.
Tech Stack: Snowflake, BigQuery, dbt, Fivetran, Airflow, Python, SQL
  • Consolidated clinical, operational and research datasets into Snowflake and BigQuery to support healthcare analytics, cohort analysis and longitudinal research reporting.
  • Built repeatable ingestion workflows using Fivetran, Airflow and Python to handle EHR schema drift, delayed feeds and source reliability issues before data reached research dashboards.
  • Developed dbt models, macros and window-function-heavy SQL with time-aware encounter logic to centralize reusable healthcare metric definitions for researchers.
  • Added freshness checks, record-count validations and profiling controls; tuned Snowflake queries and dbt lineage views to improve source-to-report traceability.
  • Documented lineage and data quality rules for research consumers and automated profiling to make cohort extraction reproducible and auditable.
  • Mentored student teams on pipeline design, code reviews and best practices for data validation and reproducible analytics workflows.
Capgemini
Hyderabad, India
Senior Software Engineer
May 2021 – Aug 2022
Worked at a technology consulting firm on Azure-based data engineering solutions, implementing lakehouse practices and secure data access for analytics teams.
Tech Stack: ADLS Gen2, Azure Data Factory, Delta Lake, Azure Databricks, Python, KQL, Azure SQL
  • Managed ADLS data zones by defining folder structures, partition standards, retention and curated layer conventions to support downstream analytics and reporting teams.
  • Optimized ADF pipelines ingesting Azure SQL data by adjusting batch sizes, trigger timing and transformation placement between ADF, Azure SQL and Databricks.
  • Implemented Delta Lake tables on ADLS with schema evolution and merge patterns to enable transactional updates while preserving existing Power BI reporting compatibility.
  • Investigated data quality issues by comparing raw ADLS files with curated Delta layers and writing SQL/KQL checks to isolate problematic pipeline runs.
  • Supported cloud data security activities including ADLS RBAC, storage key rotation and encryption practices and participated in row-level security design discussions for shared datasets.
  • Automated routine maintenance and ingestion validations with Python scripts and Databricks notebooks to reduce manual remediation and improve pipeline uptime.
Capgemini
Hyderabad, India
Software Engineer
May 2020 – May 2021
Delivered AWS-based ETL solutions at a consulting firm, building Glue and PySpark pipelines to process and curate source files into S3 for analytics.
Tech Stack: AWS Glue, PySpark, S3, AWS SDK, Step Functions, Lambda, CloudWatch, Python
  • Developed AWS Glue and PySpark ETL jobs to process CSV source files into partitioned S3 datasets for downstream querying and reporting.
  • Built Python utilities leveraging the AWS SDK to validate data, remediate exceptions and automate recurring checks across S3 and Glue workflows.
  • Configured Glue Catalog metadata and tuned crawlers to handle schema inference edge cases across changing source files and formats.
  • Integrated nightly workflows with AWS Step Functions and Lambda for orchestration and monitored data quality issues through CloudWatch logs.
  • Implemented partitioning and compression strategies to reduce query latency and manage S3 storage costs for analytical workloads.
  • Collaborated with cross-functional teams to onboard new data sources, define ingestion SLAs and agree format contracts with consumers.
Operations Analytics
Software

Education

University of Bridgeport
Master of Science in Computer Science • Bridgeport, CT • Aug 2022 – Dec 2023
CMR College of Engineering & Technology
Bachelor of Technology in Electrical & Electronics Engineering • India • Aug 2016 – Apr 2020

Certifications

Databricks Certified Data Engineer Associate — Databricks
Databricks Machine Learning Certification — Databricks
Databricks Data Analysis Certification — Databricks
Python Certification

Powered by Drivetube · Create your own profile at drivetube.ai

Everything on Drivetube

Six products, two of them free forever. Start wherever you are.

Read the docs →
  • Your free digital resume

    at drivetube.ai/in/your-name: one true standard resume with a Hiring Snapshot (visa status, expected salary, notice period, work preference, relocation), an ATS-ready PDF download and a single shareable link. Free forever; interview requests come from verified employers and your contact details stay masked until you accept.

    Documentation
  • Verified jobs, posted in the last 3 days

    verified openings crawled ATS-by-ATS from 160,000+ real company career pages across 50+ ATS platforms. Shows the true posting date from the source ATS — not when a listing was indexed — and deletes every general listing 3 days after it was actually posted. No ghost jobs, no ad-sponsored listings, no staffing reposts, no account needed.

    Documentation
  • Written by senior career writers

    human-written, ATS-optimised resumes by senior career writers, from ₹499.99 / $25.99. Available in every country, written to the destination country's own standard — a US resume, UK CV, German Lebenslauf and Indian resume are genuinely different documents. A paid service, separate from the free Drivetube Profile.

    Documentation
  • Drivetube OneFrom $39.99 / 3 mo

    Daily matched jobs, tailored resumes, or we apply for you

    Drivetube's paid service. Community sends daily Jobsheets of fresh jobs matched to your profile on WhatsApp and email, with the Drivetube Community ($39.99 for 3 months or $69.99 for 6 months; ₹999 or ₹1,899 in India). Jobhunt is one-time application credits with Community included: one credit tailors one resume and its cover letter (100 for $99 with 3 months of Community, 250 for $199 with 6 months). Let Us Apply is an add-on to Jobhunt: a human expert applies 250 times for you on your credits ($0.60 each, $150). Everything lives on one dashboard at drivetube.ai/drivetubeone.

    Documentation
  • For employers — no job postings, no applications

    hiring with no job postings and no applications. Paste your real job description and AI matches it against candidates' true standard resumes, returning ranked candidates with a match %, matched and missing skills and written reasoning. Free tier included; employers pay, candidates never do.

    Documentation
  • Documentation

    every product explained in full, with feature-by-feature comparisons against the job boards, AI apply tools, resume services and hiring platforms people actually use.

  • Playbooks

    tactical job-search guides. Each post is one named tactic with the situation it applies to, the exact steps, a copy-paste message and honest failure modes. Free, no account.

The job board covers the United States, India, United Kingdom, Canada, Europe and Australia, and Resume Writing Services are available in every country.

Pricing is identical on the web, iOS and Android — no app-store markup. Monthly Community membership is purchased in the iOS app only; the website sells annual plans. The Android app has no in-app purchase, so Android members subscribe on drivetube.ai and then sign in to the app with full access. Membership follows your account, not your device.