Detailed Profile of Mentors / Experts
Manoj Kumar Dhakad
  Bengaluru / Bangalore (KA) , India - IST (Indian Standard Time)

Senior Data Engineer , Expert in Big Data , Data Pipeline Designing, Batch Pipeline Designing, Strea ...

   7 Years

INR₹ 1800 Per Hour

   English ( Fluent ) , Hindi ( Fluent ) ,

Key Skills   : Java, Hadoop & Java, Python, Spark , Spark SQL , Spark Streaming , Kafka, Design ...More

Categories : Big Data Analytics, Big Data, Data Visualization, Algorithms, Distributed Computing, ...More

Mobile Num : Verified Num of Views : 118
Tech Services Provided By Me

    ✓ Online / WFH / Remote Work

    ✓ More than 20 hours per week

Short Bio
Work Experience:
Python Data Engineer: Adobe, (Oct-2021 to till date)
● Led the Data and ML teams for DSP (Demand Side Platform)
● Led several projects for the Data, ML and Product teams.
● Worked closely with Reporting, Analytics and Accounting teams of Advertising Platform.
● Led the Qubole to EMR migration project for ML and Data team and helped other teams.
● Helped the teams to improve Query/Job performance using Spark.
● Configured EMR cluster for adhoc Spark, Hive and Presto Jobs/Queries.
● Implemented/Improved alerting for failed jobs,data spike using Python, Slack, PagerDuty and Mail.
● Managed the job scheduling tool and improved its performance and monitoring.
● Added several features to the job scheduling platform like push data to S3, Snowflake,Hive etc.
● Enabled data pipelines for ML model related and product related dashboards.
● Enabled data pipelines for reporting on advertisement platforms.
● Enabled data pipelines to exchange data with external partners, vendors.
● Managed ML model training pipelines, model serving and model monitoring dashboards.
● Deployed new ML models on K8s cluster and automated model training and prediction.
● I have done several POCs for tools, technologies and platforms.
● Identified unused computers and storage and cleaned them.

Data Engineer 3: PayPal, (May-2021 to Oct-2021)
● Understanding and solving the current issues in existing data pipelines.
● Onboarding new tables to hadoop clusters from PayPal properties like Venmo,Xoom etc.
● Extracting data from different DBs and ingesting to hadoop cluster.
● Scheduling pipelines using crontab and uc4.
● Ingesting history data in the existing tables with or without modified schemas.
● Enabling Partitioned and Non-Partitioned hive tables to other clusters which are used by management
and reporting teams.
● Working on the existing issues in kafka-spark streaming pipelines.
● Automating the data transformations and data pipelines.

Senior Software Engineer: Freshworks, (Nov-2019 to May-2021)
● Developed and Improved Big Data Pipelines on Spark, Hive, Impala, AWS, RDS
● Refactored the existing data pipelines to improve code maintainability and performance.
● Developed the data pipeline to process feedback data for model training,monitoring and generating
the reports.
● Understood Machine Learning requirements from Data Scientist and developed data pipelines to
provide appropriate data for model training.
● Developed data archival and purging solutions to remove PII.
● Designed database schemas, classifying fields and migrating data.
● Coordinating with different Solution providers to modernize existing Data Platforms.
● Worked on POCs to evaluate different technologies and platforms that can help in modernising
existing platforms.
● Developed and conducted learning sessions on Big Data and related technologies.

Specialist Programmer: Infosys Ltd, (May-2016 to Nov-2019)

● Programming Languages - Scala, Java, Python, C, SQL, HiveQL, Shell Script
● Big Data - MapReduce, HDFS, Hbase, Sqoop, ZooKeeper, Hive, Impala, Kafka, Spark SQL, Spark
Streaming, Zeppelin, Databricks, Cloudera, UC4, Snowflake, Qubole, Presto
● Cloud Computing- AWS, Amazon EMR, Kinesis, S3, DynamoDB, RDS, EC2, Azure
● Databases – MySQL, Hbase, PostgreSQL
● IDEs- Intellij IDEA, Eclipse,Pycharm, MS Visual Studio, Jupyter Notebooks.
● Operating Systems – Windows, Linux, Mac.
● Other- Github, Bitbucket, Bamboo, Jira, Confluence, Jenkins.

Skills & Experties


Tech Skills Experience Skill Level
Java, Hadoop & Java, 4 Years Expert
Python, 5 Years Expert
Spark , Spark SQL , Spark Streaming , Kafka, Designing, 6 Years Expert
Hadoop, HDFS , MapReduce , Hive, 6 Years Expert
Redis , 2 Years Intermediate
Oozie , H2O, 2 Years Intermediate
Big data, Avro , Parquet , JSON, 6 Years Expert
Databases, Data Warehousing, NoSQL , MySQL, PostgreSQL , HBase , Cassandra , 5 Years Expert
Prometheus , Grafana , Ganglia Monitoring System, 2 Years Intermediate
Flink , Cloudera Impala , 3 Years Intermediate
Project Details
Project Title Not Specified
Start Date Not Specified End Date Not Specified
Project Details:
Employment / Work History
Company Name Designation Duration
Not Specified Not Specified Not Specified

You have Signed Out of Your Account.

Please press 'Reload' to Sign in to BigDataLogin again.

You have Signed in to BigDataLogin as a Recruiter..

Please click "Reload" to redirect to the Recruiter Dashboard.