Venkatesh Koya
Professional Summary
Data Engineer with 4 years of experience designing and delivering production-grade data platforms, distributed data pipelines, and real-time streaming systems across AWS and Azure environments. Proficient in Apache Spark, PySpark, Databricks, Kafka, Airflow, and Snowflake with a strong track record of building scalable ETL and ELT architectures, cloud data lake solutions, and Delta Lake-based data lakehouses that power enterprise analytics. Adept at migrating legacy ETL systems to modern cloud-native frameworks, optimizing Spark job performance, and driving data reliability and consistency across complex multi-source environments.
Technical Skills
Work Experience
- Designed and deployed scalable ETL pipelines using PySpark on AWS Glue and Databricks, enabling efficient large-scale batch data processing for enterprise workloads.
- Architected and maintained a cloud data lake using AWS S3, Glue Data Catalog, Athena, and Redshift to support reliable analytics ingestion and querying.
- Developed and managed Airflow DAGs to orchestrate complex scheduled and dependency-driven data pipelines, improving reliability and observability.
- Built near real-time streaming data pipelines leveraging Kafka and Spark Streaming to process and deliver time-sensitive financial data.
- Led migration of legacy MapReduce-based ETL workloads to a PySpark-based distributed processing framework, reducing processing time and operational overhead.
- Developed dbt models to standardize data transformations, enforce business logic consistency, and improve data quality across downstream pipelines.
- Built and maintained large-scale data processing pipelines using PySpark on AWS EMR to handle high-volume financial datasets with low latency.
- Designed and implemented batch and streaming ingestion pipelines using Kafka, AWS Kinesis, and Spark Streaming to support real-time analytics use cases.
- Architected a Delta Lake-based unified data lakehouse to seamlessly support both batch and streaming workloads with ACID-compliant data management.
- Automated ETL workflows using AWS Glue and Airflow, increasing pipeline reliability, reducing manual intervention, and enhancing maintainability.
- Tuned Spark job performance through advanced partitioning strategies, data caching, and execution plan optimization, resulting in significant reduction in processing time.
- Implemented CI/CD pipelines using Git, Docker, and GitHub Actions to automate deployment and improve code delivery consistency.
- Supported enterprise applications and production ETL systems, maintaining high availability and minimizing downtime across critical business processes.
- Developed and optimized SQL Server queries and database schemas, improving query performance and overall application responsiveness.
- Automated ETL workflows using SSIS packages and SQL scripting, reducing manual data processing efforts and improving pipeline throughput.
- Performed database tuning activities including indexing, query plan analysis, and performance profiling to enhance system efficiency.
- Implemented Azure cloud services including Azure App Services, Azure SQL Database, and Azure Key Vault to support secure and scalable application deployments.
- Managed incident resolution, production support, and release activities, ensuring minimal business disruption and adherence to SLA commitments.
Education
Powered by Drivetube · Create your own profile at drivetube.ai
Everything on Drivetube
Six products, two of them free forever. Start wherever you are.
Your free digital resume
at drivetube.ai/in/your-name: one true standard resume with a Hiring Snapshot (visa status, expected salary, notice period, work preference, relocation), an ATS-ready PDF download and a single shareable link. Free forever; interview requests come from verified employers and your contact details stay masked until you accept.
Documentation- Free Job BoardFree
Verified jobs, posted in the last 3 days
verified openings crawled ATS-by-ATS from 160,000+ real company career pages across 50+ ATS platforms. Shows the true posting date from the source ATS — not when a listing was indexed — and deletes every general listing 3 days after it was actually posted. No ghost jobs, no ad-sponsored listings, no staffing reposts, no account needed.
Documentation - Resume Writing ServicesFrom $25.99
Written by senior career writers
human-written, ATS-optimised resumes by senior career writers, from ₹499.99 / $25.99. Available in every country, written to the destination country's own standard — a US resume, UK CV, German Lebenslauf and Indian resume are genuinely different documents. A paid service, separate from the free Drivetube Profile.
Documentation - Drivetube OneFrom $39.99 / 3 mo
Daily matched jobs, tailored resumes, or we apply for you
Drivetube's paid service. Community sends daily Jobsheets of fresh jobs matched to your profile on WhatsApp and email, with the Drivetube Community ($39.99 for 3 months or $69.99 for 6 months; ₹999 or ₹1,899 in India). Jobhunt is one-time application credits with Community included: one credit tailors one resume and its cover letter (100 for $99 with 3 months of Community, 250 for $199 with 6 months). Let Us Apply is an add-on to Jobhunt: a human expert applies 250 times for you on your credits ($0.60 each, $150). Everything lives on one dashboard at drivetube.ai/drivetubeone.
Documentation - Drivetube HireFree tier
For employers — no job postings, no applications
hiring with no job postings and no applications. Paste your real job description and AI matches it against candidates' true standard resumes, returning ranked candidates with a match %, matched and missing skills and written reasoning. Free tier included; employers pay, candidates never do.
Documentation
- Documentation
every product explained in full, with feature-by-feature comparisons against the job boards, AI apply tools, resume services and hiring platforms people actually use.
- Playbooks
tactical job-search guides. Each post is one named tactic with the situation it applies to, the exact steps, a copy-paste message and honest failure modes. Free, no account.
The job board covers the United States, India, United Kingdom, Canada, Europe and Australia, and Resume Writing Services are available in every country.
Pricing is identical on the web, iOS and Android — no app-store markup. Monthly Community membership is purchased in the iOS app only; the website sells annual plans. The Android app has no in-app purchase, so Android members subscribe on drivetube.ai and then sign in to the app with full access. Membership follows your account, not your device.