Cloud Computing of a Batch Verification Process

Graduate Student
Wei-You Lin
Advisor
Sheng-Luen Chung
Date
October , 28, 2016

Interactive Context Laboratory

 Outline
Interactive Context Laboratory

. Intrduction
.Cloud Computation In Spark Framework
.Performance

2
 . Intrduction
Interactive Context Laboratory

Dai[1] has implemented an Automatic

Verification system (AVS) .
AVS is about Taiwanese stock market trend in
investment community.
Pending OracletEvery recommended article
will verify close prices In the futures.

[1]Yu-Jhan Dai," Automatic Verification of Stock Recommendations in Investment

Social Webs", Master's Thesis, National Taiwan University of Science and
Technology Department of Electrical Engineering,2016

3
 . Intrduction
Interactive Context Laboratory

Long term verification target Return On

Investment (ROI)
Long-term investment where ROI are checked
about 6 x 20 = 120 verifications .
To verify a batch of 500 oraclets of long-term
investment predictions are checked about 120
x 500 = 60,000 verifications.
Verification Data Source Taiwan Stock
Exchange Corporation Website

4
 .Cloud Computation In Spark Framework
Interactive Context Laboratory

Thispaper cloud computation framework is

Spark.
About Spark
Resilient Distributed Datasets-RDD
Transformation
Action
High fault tolerance
Parallelable data structure

5
 .Cloud Computation In Spark Framework
Interactive Context Laboratory

Standalone mode of Spark allocates resources

for 3 virtual machines.
Abour main components
Worker node
Worker cores
Worker memory
Repartition
Parallelism

6
 .Cloud Computation In Spark Framework
Interactive Context Laboratory

AVS become batch distributed computing

system where require changes.
SparkContext
Cluster address
The amount of split RDD files
Execution environment
Broadcast data source url
Map instruction
Outcome save to database
7
 .Performance
Interactive Context Laboratory

Our system resource have 12 cores and 12GB

of RAM.
Number of slave nodes Total of node cores Cost time
Single node 1 72m
2 nodes 2 49m
3 nodes 3 35m
3 nodes 6 25m
3 nodes 9 20m
3 nodes 12 18m
4 nodes 4 32m
5 nodes 5 30m
6 nodes 6 26m 8
 .Performance
Interactive Context Laboratory

Limit
Repartition to delay
Web Crawler speed
Our System is possible to increase the
processing speed with the help of cloud
computing .
Our system could operate not only in PTT
platform but also in all other social
media/community of investment groups.
9