Skip to content

Subramanyam G

Sr Analyst Data Engineer • Bengaluru, Karnataka • drivetube.ai/•••••

Professional Summary

Senior Data Engineer with 8.9+ years of experience designing and operating enterprise data pipelines and lakehouse solutions. Strong hands-on experience with Microsoft Fabric, Azure Data Factory and Databricks notebooks to build Bronze/Silver/Gold medallion architectures and orchestrate notebooks for automated ETL. Deep expertise in PySpark and Spark for large-scale transformations, Hive and Vertica for analytical storage, and MongoDB for dashboard-serving stores. Experienced across retail, supply chain and manufacturing domains, migrating legacy Hadoop/MapReduce jobs to Spark-based pipelines and implementing data validation and cleansing frameworks. Adept at translating business requirements into technical specifications, maintaining CI with Git/SVN, and enabling repeatable, monitored data delivery for analytics and reporting teams.

Technical Skills

Programming Language: Scala,Java
Databases: MongoDB,Vertica,Microsoft Fabric
Cloud Platforms: TIDAL,Apache Kafka,SnapLogic
Version Control & Development Tools: Git
Data Engineering & Processing: PySpark,MapReduce,HDFS
Data Warehousing: EDW,Lakehouse
Data Integration & ETL: ADF,Talend,Data Validation,ETL
Compliance & Governance: Data Quality,Data Lineage

Work Experience

DXC Technology
Bengaluru, Karnataka, India
Hadoop / IT Sr Analyst Data Engineer
Dec 2019 – May 2025
Worked on retail and contract data consolidation into a lakehouse; built pipelines and notebook orchestration to feed dashboards and downstream systems.
Tech Stack: Azure Data Factory, Microsoft Fabric, Databricks Notebooks, PySpark, Spark, Hive, MongoDB, Sqoop, Git
  • Architected and implemented lakehouse ingestion pipelines using Azure Data Factory to orchestrate Databricks and Microsoft Fabric notebooks across Bronze/Silver/Gold layers for multi-source sales and contract data.
  • Developed modular PySpark transformations in Databricks notebooks to normalize, enrich and deduplicate large contract and sales datasets for downstream analytics.
  • Optimized Hive and Spark jobs to improve staging throughput and stabilize nightly batch windows for incremental contract loads.
  • Implemented an automated data validation and cleansing framework with PySpark notebooks orchestrated by ADF pipeline activities to catch schema drift and invalid records.
  • Migrated legacy Sqoop and MapReduce ingestion flows into Spark-based pipelines and consolidated processed outputs into MongoDB to power business dashboards.
  • Partnered with business analysts to convert functional requirements into technical specifications and maintained CI and release artifacts using Git and SVN.
Birlasoft Limited
Bengaluru, Karnataka, India
Hadoop / IT Senior Software Engineer
Jan 2019 – Dec 2019
Delivered customer ID logic changes and schema rollbacks for enterprise customer master and enrichment flows in an EAP data lake environment.
Tech Stack: Scala, Spark, Hive, PySpark, Java
  • Reverted and refactored customer ID derivation logic in ConsolidatePanHPEIncremental.scala to restore pre-change behavior while preserving data integrity.
  • Developed Java utilities and Scala-based unit tests to validate scenarios where multiple party IDs map to a legacy id and to generate fallout reports.
  • Modified DDL and executed schema changes for the ib_panhpe_enr table to restore pre-customer-id fields and ensured backward compatibility for downstream consumers.
  • Automated rollback and deployment scripts and validated dataset consistency using Hive queries and PySpark where necessary.
  • Documented functional and technical design changes and updated operational runbooks to streamline release and support activities.
  • Tuned Spark job configurations and memory settings to stabilize nightly batch executions and reduce job failures.
KPIT Technologies Limited
Bengaluru, Karnataka, India
Hadoop / IT Senior Software Engineer
Jan 2018 – Jan 2019
Built consumer data pipelines and analytical layers (TIDAL/Vertica stack) to support market analytics, campaigns and predictive modelling for high-volume online business.
Tech Stack: Vertica, TIDAL, ETL
  • Designed and built pipelines to ingest and consolidate multi-source consumer data into Vertica to support analytics and campaign generation for a $28.5B online business.
  • Implemented TIDAL orchestration for complex batch workflows and optimized execution ordering to support iterative predictive modeling.
  • Built ETL transformations and aggregation layers to provide page-level measures and to enable fast ad-hoc analysis for Site Catalyst integration.
  • Developed analytical data models and schema designs in Vertica to accelerate query performance for campaign and reporting teams.
  • Created monitoring and alerting for data quality issues and automated remediation scripts to reduce manual intervention.
  • Coordinated with analytics and modeling teams to deliver feature-ready datasets for machine learning experiments and reporting.
ASAP Info Systems Private Limited
Bengaluru, Karnataka, India
Software Engineer
Mar 2015 – Dec 2015
Worked on EAP self-service data lake ingesting ERP data via integration fabric; implemented Kafka/Talend based ingestion and Hadoop processing for enterprise reporting.
Tech Stack: SnapLogic, Kafka, Talend, HDFS, Hadoop, MapReduce
  • Implemented SnapLogic-based ingestion to push SERP messages into Kafka topics as the first step of enterprise data lake intake.
  • Developed Talend jobs to consume Kafka topics and ingest cleansed ERP messages into HDFS for downstream Hadoop processing.
  • Authored Hadoop transformations to standardize ERP data and prepare datasets for self-service reporting across business functions.
  • Built operational dashboards and automated failure recovery routines for ingestion pipelines to improve pipeline availability.
  • Created data lineage capture and metadata processes to support governance and dataset discoverability within EAP.
  • Tuned MapReduce jobs and optimized cluster resource usage to improve throughput and reduce job contention during peak windows.
Systems Domain
Bengaluru, Karnataka, India
Software Technical Associate
Feb 2014 – Dec 2014
Delivered finance-focused ETL and reporting support for EG Instrumentation, extracting EDW sales and margin data to regional servers and ensuring timely executive reporting.
Tech Stack: EDW, EMR
  • Designed ETL flows to extract sales and margin data from EDW and populate regional servers (JADE, JIO, EMR) to meet quarter reporting windows.
  • Developed scheduled jobs and orchestrations to ensure timely availability of quarterly sales reports for executive review.
  • Implemented data validation checks to ensure accuracy of revenue and margin figures across AMS, APJ and EMEA regions.
  • Optimized extraction queries and partitioning strategies to accelerate end-of-quarter report generation and reduce query runtimes.
  • Documented data mappings and maintained runbooks for the EGI team to reduce recovery time during incidents.
  • Backed production incidents, performed root cause analysis and applied fixes to restore pipeline operations.

Education

Osmania University (OU)
Master of Computer Applications (MCA) • Hyderabad, Telangana, India • 2010
Krishna Devaraya University
BSc (Maths, Electronics & Computer Science) • Anantapur, Andhra Pradesh, India

Certifications

AZ-900: Microsoft Certified: Azure Fundamentals — Microsoft • 2024
AI-900: Microsoft Certified: Azure AI Fundamentals — Microsoft • 2024
AWS Certified Cloud Practitioner — Amazon • 2024
Databricks Certified Data Engineer Associate — Databricks • 2024

Powered by Drivetube · Create your own profile at drivetube.ai

Everything on Drivetube

Six products, two of them free forever. Start wherever you are.

Read the docs →
  • Your free digital resume

    at drivetube.ai/in/your-name: one true standard resume with a Hiring Snapshot (visa status, expected salary, notice period, work preference, relocation), an ATS-ready PDF download and a single shareable link. Free forever; interview requests come from verified employers and your contact details stay masked until you accept.

    Documentation
  • Verified jobs, posted in the last 3 days

    verified openings crawled ATS-by-ATS from 160,000+ real company career pages across 50+ ATS platforms. Shows the true posting date from the source ATS — not when a listing was indexed — and deletes every general listing 3 days after it was actually posted. No ghost jobs, no ad-sponsored listings, no staffing reposts, no account needed.

    Documentation
  • Written by senior career writers

    human-written, ATS-optimised resumes by senior career writers, from ₹499.99 / $25.99. Available in every country, written to the destination country's own standard — a US resume, UK CV, German Lebenslauf and Indian resume are genuinely different documents. A paid service, separate from the free Drivetube Profile.

    Documentation
  • Drivetube OneFrom $39.99 / 3 mo

    Daily matched jobs, tailored resumes, or we apply for you

    Drivetube's paid service. Community sends daily Jobsheets of fresh jobs matched to your profile on WhatsApp and email, with the Drivetube Community ($39.99 for 3 months or $69.99 for 6 months; ₹999 or ₹1,899 in India). Jobhunt is one-time application credits with Community included: one credit tailors one resume and its cover letter (100 for $99 with 3 months of Community, 250 for $199 with 6 months). Let Us Apply is an add-on to Jobhunt: a human expert applies 250 times for you on your credits ($0.60 each, $150). Everything lives on one dashboard at drivetube.ai/drivetubeone.

    Documentation
  • For employers — no job postings, no applications

    hiring with no job postings and no applications. Paste your real job description and AI matches it against candidates' true standard resumes, returning ranked candidates with a match %, matched and missing skills and written reasoning. Free tier included; employers pay, candidates never do.

    Documentation
  • Documentation

    every product explained in full, with feature-by-feature comparisons against the job boards, AI apply tools, resume services and hiring platforms people actually use.

  • Playbooks

    tactical job-search guides. Each post is one named tactic with the situation it applies to, the exact steps, a copy-paste message and honest failure modes. Free, no account.

The job board covers the United States, India, United Kingdom, Canada, Europe and Australia, and Resume Writing Services are available in every country.

Pricing is identical on the web, iOS and Android — no app-store markup. Monthly Community membership is purchased in the iOS app only; the website sells annual plans. The Android app has no in-app purchase, so Android members subscribe on drivetube.ai and then sign in to the app with full access. Membership follows your account, not your device.