Installing Spark On Apache Hadoop Cluster

​
 

​

In this blog post, we will look into installing Spark on Hadoop cluster. Below is a step-by-step guide for the entire installation procedure. 

1. Change directory to hadoop installation directory. In my case, I have installed hadoop in /opt/local/hadoop
   $cd /opt/local/hadoop
2. I have hadoop 3.2.1 is installed. I have downloaded ths spark 2.4.5 version $ wget

https://downloads.apache.org/spark/spark-2.4.5/spark-2.4.5-bin-hadoop2.7.tgz

Please verify whether the Spark version supported your hadoop installation version or not.

1. Install scala on all the nodes $sudo apt install scala
   $scala -version

1. Extract spark and rename
   

   $ tar -xvzf spark-2.4.5-bin-hadoop2.7.tgz
   
   $ mv spark-2.4.5-bin-hadoop2.7 spark

1. Set Spark_Home in environment variable

   $cd
   $vim ~/.basrc
   Add following line in bashrc
   export SPARK_HOME=/opt/local/hadoop/spark export PATH=$PATH:$SPARK_HOME/bin

1. source ~/.bashrc
2. cd $SPARK_HOME/conf
3. Configuring spark-env.sh file
   mv spark-env.sh.template spark-env.sh
   Edit spark-env.sh and add below line
   export SPARK_MASTER_HOST=server1.bigdata.com export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
4. Configuring spark-defaults.conf
   mv spark-defaults.conf.template spark-defaults.conf
   Edit and add below lines in spark-defaults.conf
   spark.master yarn spark.yarn.am.memory 768m

spark.driver.memory 512m
spark.executor.memory 512m
spark.eventLog.enabled true
spark.eventLog.dir hdfs://server1.bigdata.com:9000/sparklogs

Below is the explanation for each of the variable configured in the spark-defaults.conf

1. “Spark.master yarn”
   This will set spark ready to interact with yarn
2. “spark.yarn.am.memory 768m”
    

This will set maximum memory allocation allowed to the yarn container. In this case we have set it to 768 megabytes. If the memory requested will be more than the maximum, yarn will reject the creation of the container and spark application will fail to start.

The value of yarn memory memory application must be lower than that of yarn.scheduler.maximum-allocation-mb in $HADOOP_CONF_DIR/yarn-site.xml.

c. “Spark.driver.memory 512m”
Set 512 megabytes default memory allocated to Spark Driver in cluster mode In cluster mode, the Spark driver runs inside YARN Application master. The amount of memory requested by spark at initialization is configured.

d. “spark.executor.memory 512m”
Set the Spark Executor base memory to 512 megabytes
The Spak Executors allocation is calculated based on two parameter inside $SPARK_HOME/conf/spark-defaults.conf

• ?  spark.executor.memory: sets the base memory used in the calculation
• ?  spark.yarn.executor.memoryOverhead: is added to the base memory,
  It defaults to 7% of base memory, with a minimum of 384 MB
  Example: for spark.executor.memory of 1Gb , the required memory is 1024+384=1408MB. For 512MB, the required memory will be 512+384=896MB

???????

e. To enable Spark job logs in hdfs add in spark-defaults.conf spark.eventLog.enabled true
spark.eventLog.dir hdfs://server1.bigdata.com:9000/sparklogs

Make sure to create “sparklogs” directory in the hdfs. $hdfs dfs -mkdir /sparklogs

10.Configuring salves

mv slaves.template slaves

Edit and below line slaves file

server1.bigdata.com server2.bigdata.com server3.bigdata.com

11. Copy the spark directory on slaves node.
Create spark directory in hadoop home directory on both the slave nodes server1.bigdata.com and server2.bigdata.com from hadoopuser

Below command on master node will copy the spark configuration and other detail on slave nodes:

$scp -r /opt/local/hadoop/spark/* h?adoopuser@server2.bigdata.com?:/opt/local/hadoop

$scp -r /opt/local/hadoop/spakr/* hadoopuser@server3.bigdata.com?:/opt/local/hadoop

Congratulations! You have installed Spark on Apache Hadoop cluster. Change the directory to SPARK_HOME
$cd $SPARK_HOME

Run below command
$./bin/spark-submit --class org.apache.spark.examples.SparkPi --master yarn-client --deploy-mode client --driver-memory 1024m --executor-memory 512m --executor-cores 1 examples/jars/spark-examples_2.11-2.4.5.jar
400

During the job execution you can check the status on spark web UI

http://server1.bigdata.com:4040/jobs/

​

​

We are a SaaS Application Development Company that provides end-to-end web and mobile app development services with a focus on next-gen technologies. Our development team is skilled at using the latest tools, frameworks, and SDKs to build performance-driven web and mobile applications that are easy to scale and customize. We have successfully completed several full-fledged SaaS projects for clients from across the globe. Our 360-degree SaaS application development services include quality assurance and software testing services.  

#SaaSApplicationDevelopment  #mobileappdevelopment 

​

Leanback Library For Android Smart TV Application Development

In this blog post, we'll study how to easily enable a smart TV application supported by Android TV using the Leanback library. At the end of the blog, we can expect to have a UX compliant single APK for Android TV that aids in smart TV application development.

Overview of The Lean-back Concept

To make easy integration for developers android created the Leanback library. The core fragments you'll be working with are:

BrowseSupportFragment - For browse through a video player with a complete video library

PlaybackOverlayFragment - For getting the full control of video playback

DetailsFragment - To Display the details information of a particular video

All these fragments are using the Model View Presenter pattern. We'll bind our data model to the view using presenter class.

Browse Support Fragment

In this step, you'll learn about the framework for the video browse support fragment. The following concepts to take away in this step are:

a). Update the android manifest for Android TV

b). Extends our fragment with the Leanback Browse Support Fragment

Add these Lean-back dependencies to the build Gradle file. and synchronize these;

  implementation 'androidx.leanback:leanback:1.0.0'

  implementation 'androidx.appcompat:appcompat:1.1.0'

For an application that requires support for old versions of Android, we should make sure that the path of code utilizing libraries with a higher minimum SDK does not run on devices with version < minimum SDK (library).

Now, create a browsing activity.

In the Android Studio project, create a new project with package name com.android.example.leanback by right-clicking the folder and clicking New -> Package.

Name the new package lean back demo.

source: Google Codelabs

Creates a new Blank Activity name as Leanback Activity, and click on Finish;

source: Google Codelabs

If we want to use lean-back for the android smart TV then we should declare the launcher activity for Android TV. and add an intent filter to the LEANBACK ACTIVITY tag. that is android.intent.category.LEANBACK_LAUNCHER which told to Android TV to launch LeanbackActivity when the application is on the running state.

and also we should declare the lean back tag in the manifest;

For handling unsupported hardware features in the android smart TV

Some features are not available on smart TV, We need to explore their requirement. If We use any of the following features we'll need to add the android: required="false" to the manifest. In the following image we'll under the about it;

The Browse Support Fragment class in the android Leanback library which allows us to create a primary layout for browsing categories as well as rows of media items with a minimum amount of code.

                                                   Browse Support Fragment with combination of Row Fragment and Side Navigation Categories

Details Fragment

The Details Fragment includes classes for displaying additional information about the description, reviews, and media items and for taking action on that item, such as purchasing it or playing its content etc. We can see the example of details fragment in the following picture;

source: Medium

For every application, especially for smart TV application development, UI is one of the biggest differences between the Android smart TV application. We need to make UI suitable for smart TV usage. Because the user uses a remote control and cannot use the “touch screen” like mobile function with a smart TV. To achieve this requirement, the Leanback library plays a major role in android smart TV application and it has multiple widgets and features that are better for the developers as they can easily implement UI which satisfies these requirements and thus suitable for TV usage.

Engaging More users with Live Streaming Apps

According to the Interactive Advertising Bureau Survey around 47% of live video streaming viewers across the globe are streaming more live videos in comparison to the last year. 

It is evident through the above stats that the demand for live streaming apps is growing rapidly. Various industries are using live video streaming apps to establish their business and connect with users from different parts of the globe. As a video streaming app development company we receive requests for live video streaming apps that can be used for sharing knowledge or imparting knowledge to students in remote areas. We also receive requests for OTT app development services that users can use to consume interactive video content any time anywhere. 

Also Read: Everything You Need To Know About OTT App Development 

According to eMarketers research consumers on average spend 83 minutes per day consuming digital video content. It is expected that the average time on these streaming apps may rise up from 83 minutes to 92 minutes by the end of 2020. 

Facebook Live, YouTube Live, Twitch, Instagram Live are the popular live streaming applications that have gained huge traction over the years. Investing in a live video streaming app can be really beneficial for businesses that are planning to establish their business in other countries. With the help of live video streaming apps businesses will be able to spread their links overseas. 

Let’s explore different types of Video Streaming apps

 

Live Broadcasting App

Live broadcasting apps enable users to go live and interact with people. These apps entertain users and allow them to watch videos in real-time mode. Live videos are broadcasted and recorded simultaneously. The recording mode enables users to watch a live video even after it is finished. Various YouTubers use YouTube’s live mode to communicate live with their audiences. The youtube channels are now collecting details about their customer's preferences through Live streaming. YouTubers have started live voting to get views from their audiences on different topics. 

Facebook Live is used for sharing experiences with the audiences. The customer category of YouTube audiences differs from Facebook live. Before you decide to invest in a live video streaming app it is good you decide what purpose you should fulfill. 

 

Video On Demand Streaming 

Video streaming apps like Netflix have gained huge popularity in recent years. These apps enable users to consume content at any time of their preference. The main reason for the popularity of video streaming apps is new and unique content available on these platforms. Video streaming apps are available for smart TV as well as for mobile phones to add comfort to consumer viewing experiences. Using video streaming apps consumers can watch the latest movies, web series, comedy shows, and much more. The users have to pay a subscription fee to consume content through these video streaming apps. The leaders in this niche are Netflix, Hulu, Vevo, Amazon Prime, etc. 

Understanding the Benefits Of Creating Live Streaming Application

Investing in a live streaming app can be really beneficial for businesses that are planning to establish their business overseas. Let’s explore the benefits of getting a live video streaming application developed:  

Gain More Attention Of Customers 

Live streaming enables businesses to establish a strong connection with their customers. YouTuber’s use live streaming options because during that time they can better understand their audiences and establish a better connection with them. At the time of live streaming, businesses gain more attention. Also, at the time of live streaming more users connect online than offline. Live streaming apps keeps customers engaged and gain more attention from them.   

Increase Brand Loyalty 

At the time of live streaming, a huge audience gets connected with you that helps in establishing your brand name. Businesses can use live video streaming app to market products that customers are looking for and increase their loyalty towards your brand. 

Huge Return On Investment

Just by rendering video content through live streaming apps businesses can earn huge profits. To make huge returns on your investment ensure the video quality is exceptional and the content is highly engaging.  

Are you planning to invest in a video streaming app like Netflix or a Live streaming app like YouTube and Facebook Live? We are a video streaming app development company that video streaming apps for businesses across the globe. Our popular video streaming app development solutions include: 

• Video-on-Demand Apps
• Subscription-Based Video Streaming Apps
• Advertisement Based Video Streaming apps

For queries on video streaming app development, get in touch with our experts today! 

Chatbot development company

Oodles Technologies is an offshore Chatbot Development Company in India that offers top-of-the-line Chatbot solutions to the businesses at the best market price. With a seasoned team of developers, we strive to deliver best-in-class Chatbot Development Services tailored to the requirements of our clients.

We, at Oodles, are pioneers in building chatbots that help save your time and money. Oodles Technologies is helping companies to conceptualize its business globally. We give highly intelligent chatbot solutions that are topped with artificial intelligence and advanced algorithms.

5 Reason to Choose Kotlin for Android App Development

Designing an application becomes completely meaningless if it is unable to function in the desired manner. Various mobile applications face downtime due to the programming language which has been used in its processing. Various app developers had been searching for a programming language that helps them overcome the difficulties they face developing an application. To help software developers with designing a perfect application JetBrains has introduced Kotlin - a programming language that can help in building commercial applications without any hassle. Let's move ahead and know why developers should use Kotlin for ANDROID APP DEVELOPMENT.

1. Kotlin Offers Brevity

Earlier mobile app developers were dependent on Java programming language to get any kind of application developed But with time they searched for a language which is precise and offers brevity. Kotlin is the perfect programming language that not only offers brevity but also ensures there are lesser errors in code. With lesser errors the production becomes faster, as a result, the application performs better.

2. Open Source

Just like Java, Kotlin is also an open source project. Since it is an open source language the developers can easily expect high-quality development. Apart from this Kotlin also offers support from the slack team through which the chances for errors becomes negligible.

3. Language With a Great IDE Support

In the testing phases of Kotlin, it has gone through several stages of alpha and beta improvements. It was used in a lot of projects before its final release so that the developers do not face any problem while developing an application with it. With world class IDE support it ensures smooth operation and features that are similar to Java.

Read more at - 5 Reason to Choose Kotlin for Android App Development

Cloud application development services

With our cloud application development services, access your applications and support business functionalities with cloud. At Oodles, we provide custom cloud solutions that are apt for your business and ensure that your business is no more limited with fixed costs or finite infrastructure. We focus on elaborating scalability and security at the same time, that matches with your business dynamics. Ready to make your business dynamic?

Plug into cloud with our core services

Make your business smart and futuristic with cloud application development services

Cloud Consulting and Application Development

Our cloud experts analyze your business, functional, and technical requirements and work hand-in-hand with your team to provide business-relevant cloud centric implementation strategies and solutions. We ensure that your cloud-enabled applications aim to drive benefits such as flexibility and agility at reduced costs.

Cloud Architecture Services

Our cloud architects analyze the existing infrastructure and architect or re-architect your system, assist you to choose the right cloud platform to deliver high performance, data sensitiveness, compliance to industry standards, and futuristic solutions.

Cloud Integration Services

We enable our client's to take advantage of economies of scale through Cloud, module, data, and application integration services. We add value to your IT management system with following integrations and more:

• SaaS Application Integration Providing SaaS integration services for Salesforce, HubSpot, Microsoft, Kaltura SaaS, and more
• PaaS Application IntegrationDelivering PaaS integration services such as MongoDB cloud, Azure ML, AWS machine learning, and so forth
• FaaS Application IntegrationProviding serverless computing services such as AWS lambda, Kubernetes, Azure, Google cloud functions, etc

Cloud Migration Services

Our end-to-end approach handles cloud migration complexities and creates a scalable IT environment for your organization. We make migrations risk-free from on-premise to cloud environment (public, private, or hybrid) and cloud to cloud. We support tight migration, minimize downtime with live capture and incremental data sync, ensure there is no lock-in issues and thus reduce time and investment cost.

AI Application Development Services

At Oodles, we offer AI driven solutions, services, and technologies to make your applications smarter, understand your customers better, and be more customer-centric. The advanced analytics and algorithms powered by AI utilize the generated data through IoT, mobile apps, and/or the websites and enable you to make informed decisions for enhancing customer experience.

Recommendation Engine

We help you sell more online through data-powered user engagement. Our high-octane recommendation engines features enable you to capture the customer behavior and explore their preferences. This helps with recommending relevant products, improve their experience with your brand, increase footfall, and thereby bring high ROI.

Chatbot

We leverage our years of business experience and Machine Learning capabilities to create intelligent chatbots for your business. Our chatbot development services bring the tangible change a business needs by helping you reach more customers and serve them better.

Computer Vision

We let you spruce up your application with the revolutionary aspects of Computer Vision. We make use of the smart OpenCV algorithms and an across-the-board AI toolkit to provide an extraordinary user experience for the end-users.

Natural Language Processing Applications

Implement Natural Language Processing technology in your organization to derive understanding from unstructured data generated from customer conversation through social media, online and call logs.

Predictive Analytics

Our predictive analytics applications and services help you use tools and techniques to reach the customers faster. Gain the competitive edge by forecasting demand estimates, determining market size, forecasting new product growth, projecting and predicting stock prices, adopting pricing strategies, optimizing marketing campaigns, reducing risks, and preventing customer loss.

Resource Planning Solutions

Use our AI-based planning solutions to streamline the workflow of your business. Gain an efficient automated planning with our planning algorithms for explainability (autonomous driving), rapid prototyping, of-the-shelf planners, tweaking of the existing model(s), and more.

Read more at - AI Application Development Services

OTT Video Solutions Development

Oodles Technologies sets the benchmark in the IT ecosystem by creating dynamic and user-friendly online video content platform. We assist you in organizing and managing the online video content efficiently with our advanced video solutions. Our in-depth expertise and proven methodologies in online video content management systems facilitate you to deliver rich and engaging offerings at scale.

Video-on-demand and Live Streaming

Being enthusiastic professionals to adopt fast technologies, we deliver video-on-demand and Live streaming services to increase brand awareness, audience engagement, followers, and potential customers.

Implementing video-on-demand and live streaming helps you to envelop all kinds of customers and charge your business. We assist you in sharing updates straight from the table through live streaming and thus create more buzz. In addition, we help you catch the customers at their convenient time through video-on-demand services.

Read more at - Digital Video OTT Solutions,OTT Video Delivery Solution,Video Streaming App Development

MEANJS Application Development Services

Oodles Technologies is emerging as a leading offshore software development services provider in India that offers end-to-end MEANJS Application Development Services to the clients. We have carved a niche in the web development industry. With several years of experience under our belt, our highly skilled MEAN Stack Developers have transformed themselves to work both on frontend and backend operations. We offer a real-world experience in interconnecting business processes with web development technology and deliver the best-in-class MEANJS Web Development services at the best market price.   

Blockchain App Development Services

We, at Oodles Technologies, provide best-in-class Blockchain App Development Services tailored to the requirements of our clients.

Read more info at - https://www.oodlestechnologies.com/blockchain-app-development-services

Java Development Company India

Oodles Technologies is a prominent Java Development Company in indian subcontinent and has carved a niche in the domain, offering business oriented and customer-centric Java Application Development Services to the industries across the globe.

Read more info at - https://www.oodlestechnologies.com/java-development-company

Live streaming using FFMPEG

Want to integrate your application with live streaming using FFMPEG?

Your quest ends here! Oodles Technologies is a leading company which provides live streaming services using FFMPEG. We deliver high-quality FFmpeg Development Services. Our expertise in the domain helps you to deliver quality FFmpeg Live streaming services.

For any of business-centric requirement, feel free to Contact Us.

Android Application Development Experts

Want to create a user-friendly Android App for your business? You are at the right place. With a professional team of Android Application Development Experts, we offer tech-driven solutions for all your business needs. We have the proficiency to cater your needs efficiently. Oodles Technologies is a renowned name in the field of app development, our services include up-to-date, seamlessly integrated apps.

Android is an invaluable endowment of Google to the worldwide smartphone community. Based on a Linux kernel version, it is the world’s most popular and the most extensively used mobile operating system. Being an open-source mobile operating system, Android powers an enormously large number of smartphones from different manufacturers. On contrary to this, all the other operating systems such as iOS, Blackberry, and Windows are exclusively built for their own smartphones.

 

Android was built for all the touchscreen-enabled smartphones and tablets but now it supports many other types of devices including Smart TVs, Smart Watches and Android Auto. Ever since the launch of Android, Google has released a number of system updates. There have been 7 major updates of Android so far and each one brought new features and enhancements. The most recent update to Android is Nougat 7.1 which offers some outstanding features like quick app switching and split-screen view.

Why Choose Oodles Technologies?

  

We at Oodles Technologies are pioneers in building avant-garde apps for mobile platforms like Android and iOS. With vast experience in android application development, we provide high-quality application development services at the best market price. We have a team of Android App Developers with hands-on experience in building high octane applications. Besides Android, we also have high competency in iOS App Development and we also provide app and channel development services for the popular streaming media players such as Roku, Apple TV, and Amazon Fire TV.