Kowshik Gadiyaram
Senior Data Engineer • San Ramon, CA • k***************@gmail.com • +13******151 • drivetube.ai/•••••
Professional Summary
Senior Data Engineer with 7+ years of experience designing, building, and optimizing scalable data platforms across fintech, e-commerce, industrial analytics, telecom, and enterprise domains. Proven expertise in architecting cloud-native data pipelines using AWS (S3, Glue, Lambda, EMR), Spark, Kafka, Airflow, dbt, Snowflake, and PostgreSQL to deliver reliable, analytics-ready datasets. Skilled in migrating manual, multi-team reporting processes into fully automated cloud data platforms, with strong background in ELT/ETL design, dimensional modeling, data quality frameworks, and performance optimization. Adept at collaborating with product managers, data scientists, and business stakeholders to translate requirements into governed, secure, and cost-efficient data solutions, while mentoring engineers and driving platform scalability and operational excellence.
Technical Skills
Programming Languages: Python,Shell Scripting
Databases: SQL,Snowflake,PostgreSQL,Parquet,S3 data lake,Delta tables
Cloud and DevOps: AWS S3,AWS Glue,AWS Lambda,AWS Step Functions,AWS DMS,AWS CloudWatch,IAM,Kubernetes,Monitoring and alerting,Pipeline reliability practices
Testing: Great Expectations,Custom validation frameworks,Data cleansing
Data and Analytics: Apache Spark,Databricks,EMR,Amazon Redshift,Delta Lake,Dimensional Modeling
Skills: PySpark
ETL and ELT & Orchestration: Airflow,dbt,AWS Glue Workflows,Databricks Jobs
Streaming & Messaging: Apache Kafka,Event-driven ingestion
Integration & Source Systems: Microsoft Dynamics 365 CRM,Multi-channel marketplace data ingestion
Collaboration & Productivity: Design documentation,Code reviews,Mentorship
Work Experience
AIGentics
United States
Senior Data Engineer
06/2024 – Present
Consulting on fraud detection and analytical engineering for Intuit; built batch and streaming pipelines to process credit card transaction and payments data for risk and analytics teams.
Tech Stack: Spark, Databricks, Delta Lake, Snowflake, Airflow, dbt, Kafka, AWS Lambda, EMR, Great Expectations, Kubernetes, IAM
- Designed and implemented scalable batch and streaming pipelines using Spark, Airflow, and AWS EMR to ingest multi-terabyte daily credit card transaction and payments data into a curated Delta Lake, improving downstream fraud detection and analytics latency by 35%.
- Built Databricks-to-Snowflake serving layer to load curated transaction and chargeback datasets into Snowflake, optimizing clustering and query pruning to reduce fraud investigation query runtimes and cut compute costs by ~28%.
- Orchestrated end-to-end ELT workflows with dbt and Airflow to standardize transformations across transaction, chargeback, and payments datasets, delivering consistent, governed metrics for fraud reporting and risk analysis.
- Engineered near-real-time ingestion using Kafka and AWS Lambda to stream transaction events into Databricks/Delta Lake, enabling lower-latency fraud signals for analytical engineering and investigator workflows.
- Introduced Great Expectations and custom validation checks at the ingestion layer to enforce schema and business rules on financial datasets, increasing trusted dataset adoption across fraud and analytics teams.
- Hardened data security and governance across Databricks and Snowflake with IAM, encryption, and row-level access controls; containerized pipeline components with Kubernetes to improve scalability and reliability.
Governors State University
Chicago, IL
Data Engineer
12/2023 – 05/2024
Academic capstone for Mi-Jack products converting industrial telemetry and machine logs into analytics-ready datasets to support predictive maintenance and reliability analysis.
Tech Stack: AWS S3, AWS Glue, AWS Lambda, PostgreSQL, Python, Fivetran, Airbyte, CDC
- Architected an AWS ingestion layer using S3, Glue, and Lambda to collect industrial telemetry and unstructured machine logs, enabling downstream analytics-ready processing for Mi-Jack engineering teams.
- Developed end-to-end ELT pipelines in Python and SQL to parse and transform unstructured logs into dimensional models following Kimball methodology in PostgreSQL to support predictive maintenance KPIs.
- Implemented CDC patterns to capture and ingest database changes efficiently, ensuring near-real-time consistency between operational systems and the analytics warehouse.
- Used Fivetran and Airbyte to standardize and accelerate source integrations, reducing onboarding time for new telemetry streams and increasing data availability for analysis.
- Optimized PostgreSQL query plans and indexing to handle growing telemetry volumes, cutting dashboard and analysis query response times by over 30%.
- Collaborated with faculty researchers and domain engineers to align data models with equipment behavior, improving interpretability and adoption of analytics outputs by maintenance teams.
Tech Mahindra
India
Data Engineer
07/2020 – 08/2022
E-commerce data platform project consolidating multi-channel sales, CRM, and operational data into a unified AWS-based analytics environment to support reporting and forecasting.
Tech Stack: PySpark, AWS Glue, AWS DMS, AWS Lambda, AWS Step Functions, Amazon Redshift, Amazon Athena, S3, CloudWatch, Parquet, Microsoft Dynamics 365
- Led migration of legacy spreadsheet-driven reporting to an automated AWS data platform using AWS DMS and Glue; refactored PySpark transformations to eliminate ~20 weekly manual consolidation hours.
- Engineered ETL pipelines with PySpark on AWS Glue and Lambda, orchestrated via AWS Step Functions, to ingest and transform Microsoft Dynamics 365 CRM customer and sales data into S3, reducing data refresh time from >24 hours to under 2 hours.
- Consolidated multi-channel sales data from Flipkart, Amazon, and D2C platforms using PySpark DataFrames, standardizing schemas into Amazon Redshift and cutting cross-platform reporting effort by ~60%.
- Designed and implemented a star-schema dimensional model in Redshift to integrate Sales, Inventory, Procurement, and Production data, enabling cross-functional visibility for demand and production planning.
- Built a partitioned S3 data lake with raw and curated zones in Parquet format and enabled ad-hoc querying via Athena to support historical analysis without full warehouse loads.
- Implemented monitoring with CloudWatch and Step Functions retries/alerts to maintain 99%+ pipeline reliability and reduced processing time by ~30% through performance tuning and partitioning strategies.
IBM
India
Associate Data Engineer
06/2017 – 07/2020
Enterprise data engineering supporting centralized data warehouses and reporting for internal analytics and operational teams.
Tech Stack: SQL, Python, Shell Scripting, Databricks, ETL workflows, SCD Type 2
- Developed ETL workflows using SQL, Python, and shell scripting to ingest and process transactional and operational datasets into centralized warehouses, meeting daily reporting SLAs.
- Maintained and monitored data pipelines to ensure data availability and completeness for downstream analytics consumers, resolving failures to meet SLA requirements.
- Executed data cleansing and transformation logic to standardize inconsistent source data, improving reporting accuracy across departments.
- Optimized SQL queries and indexing to improve report performance, reducing execution times by up to 20% through query refactoring and indexing strategies.
- Designed and implemented SCD Type 2 patterns to accurately capture historical dimensional changes and ensure integrity of time-variant reporting.
- Supported Databricks-managed dataset validation and performance checks, and automated routine data loads and validations to increase operational efficiency for the data team.
Education
Governors State University
Masters: Business Analytics • Chicago, USA • 05/2024
Powered by Drivetube · Create your own profile at drivetube.ai
Explore Drivetube
- Drivetube Profile — your free digital resume — at drivetube.ai/in/your-name: one true standard resume with a Hiring Snapshot (visa status, expected salary, notice period, work preference, relocation), an ATS-ready PDF download and a single shareable link. Free forever; interview requests come from verified employers and your contact details stay masked until you accept. Documentation.
- Free Job Board — verified openings crawled ATS-by-ATS from 100,000+ real company career pages across 35 ATS platforms. Shows the true posting date from the source ATS — not when a listing was indexed — and deletes every general listing 3 days after it was actually posted. No ghost jobs, no ad-sponsored listings, no staffing reposts, no account needed. Documentation.
- Job Hunt Program — managed job hunting, a one-time purchase from $199.99. JobScout matches verified roles to your real experience band, Blend AI writes a uniquely tailored resume and cover letter for every application, and the Autofill extension fills the form — or Let Us Apply submits it for you. Documentation.
- Resume Writing Services — human-written, ATS-optimised resumes by senior career writers, from ₹499.99 / $25.99. Available in every country, written to the destination country's own standard — a US resume, UK CV, German Lebenslauf and Indian resume are genuinely different documents. A paid service, separate from the free Drivetube Profile. Documentation.
- Community Membership — from $4.99/month (₹1,999/year in India). Unlocks the gated job-board filters — visa sponsorship, security clearance, workplace and application time — plus Job-Scout AI matching, Resume Report AI, Interview AI prep sheets, Recruiter Outreach AI sent from your own Gmail, Apply or Skip triage, a daily market feed and a $10,000+ library including 23 ATS-validated resume templates. Documentation.
- Drivetube Hire — for employers — hiring with no job postings and no applications. Paste your real job description and AI matches it against candidates' true standard resumes, returning ranked candidates with a match %, matched and missing skills and written reasoning. Free tier included; employers pay, candidates never do. Documentation.
Full product documentation — every product explained, with feature-by-feature comparisons against the job boards, AI apply tools, resume services and hiring platforms people actually use.
The job board covers the United States, India, United Kingdom, Canada, Europe and Australia, and Resume Writing Services are available in every country.