Simulation for Training and Testing Intelligent
Systems
Daniel W. Carruth*
* Center for Advanced Vehicular Systems, Mississippi State University, Starkville, MS, USA
dwc2@cavs.msstate.edu
Abstract— Intelligent systems require sizable sets of known
data for training. Developing datasets from real-world data
is a time-consuming and cumbersome process. Simulation
provides an alternative with key advantages: rapid data
collection, automated semantic tagging, lower cost, etc. at
the cost of realism. The major automotive autonomy efforts
rely on simulation to supplement real-world driving with
millions of miles of simulated driving. Simulators are also
being used to develop and test autonomous algorithms for
do-it-yourself autonomous remote-control cars, small-to-
medium sized unmanned ground vehicles, and intelligent
agents that exist only in virtual environments. Building and
using simulated environments for training and testing
intelligent systems provides many significant advantages,
users must be aware of the limitations and constraints of
simulated environments and carefully evaluate transfer of
learning from simulated datasets to real-world
performance.
I. INTRODUCTION
Intelligent systems are being increasingly applied in
practical ways to many domains. Of particular interest is
the growing application to robotics and self-driving
vehicles. Effectively training machine learning systems to
perform tasks requires large labeled datasets – the more
complex the task, the larger the required dataset. In
training robots to perform object grasping, one example
dataset contains 9.4 million example images [1].
ImageNET, a labeled image database, contains over 14
million images [2]. Generating large labeled datasets is
expensive, time-consuming, and cumbersome. Imagery
must be collected, semantic classes defined, annotations
made manually and validated. For the Cityscapes dataset,
it was estimated that fully annotating each image required
approximately 90 minutes [3].
There are tasks that do not require time-intensive
labeling of data. For example, an end-to-end neural
network for self-driving vehicles (e.g., [4]) requires a
large dataset of images annotated with steering angles. In
this case, a training dataset can be generated by recording
camera data and steering angle data as a human driver
operates the vehicle. While effective for training a ML
model to drive on the same track, such a dataset is limited
by the variability of the sites at which the data is collected,
the behaviors of the driver, and other factors that limit the
generalizability of the trained model.
In addition to training systems, we must also have
facilities to test the systems. Effective training and testing
depends on the ability to present the system with
representative scenarios that capture the range of scenarios
that the system may observe in the field. In particular, the
101
range of scenarios that threaten the integrity of the system
and the safety of people in the area of operation. Testing is
also critical to the software development process.
There are other factors that limit our ability to develop,
train, and test intelligent systems with real-world data:
First, real world testing of field robots or self-driving
vehicles requires that developers and test teams have
access to real world devices. Given the cost of physical
robots, and even their constituent parts, the cost of
physical testing can be prohibitive. Second, real world
testing requires physical test sites and instrumentation for
collecting and evaluating performance. In addition, these
test sites need to provide the full variety of environments,
scenarios, stimuli, hardware, etc. to which the intelligent
system is being trained to respond.
Simulation is appealing as a potential solution for these
challenges. In simulation, we can automatically annotate
sensor data. A simulation could generate hundreds of
thousands of simulated sensor data matched with fully
annotated ground truth. Human-in-the-loop simulation can
be used to generate training datasets that rely on human
data. Simulations can also use algorithms that run slower
than real-time to generate datasets to train real-time
systems. In simulation, we can randomly generate new
environments and stimuli with which to train and test the
models. We can store tests to benchmark systems or
perform regular testing of software builds. Simulations
can allow us to rapidly modify and evaluate system
parameters that would be difficult to change in the field
(wheelbase, camera position, field of view, etc.).
While simulation is widely used for training and testing
robotic systems, mobile robot platforms, self-driving
vehicles, and many other intelligent systems, simulation
will never be a perfect match to the real world. The
transition from simulation to real world performance will
be constrained by general and specific limitations of
simulation.
II. RELATED WORK
A. Real-time Training and Testing
The DARPA Robotics Challenge (DRC) and the
DARPA Virtual Robotics Challenge (VRC) was a high-
profile competition that used simulation to give teams
platforms to develop and test for a common platform. The
DRC and VRC were designed to drive innovation in
development of robotic systems, particularly humanoid
systems, for disaster response [5]. Six teams used Boston
Dynamics Atlas robots [6] in the challenge. Atlas robot
hardware was not widely available to the teams.
 An Open Source Robotics Foundation (OSRF)
simulator provided a development testbed for teams with
access to an Atlas and an opportunity to compete for
teams without access to an Atlas [7]. One of the Atlas
teams, the Florida Institute of Human & Machine
Cognition team, finished first in the Virtual Robotics
Challenge and second in the DARPA Robotics Challenge
[8].
In the transition from simulation to hardware, the team
identified challenges related to sensing, controls, and
communication that required significant revisions to their
system [8]. In particular, the simulation provided perfect,
synchronized data without the noise, distortion, and error
found in real sensor data. Similarly, the physical
parameters and controllers in the simulation did not
perfectly match the physical robot. A particular issue was
the absence of latency and communication issues in the
simulation. Even when incorporating latency in
simulation, it can be difficult to capture fully the effects
on sensor quality and operator perception and response
[9].
For the upcoming DARPA Subterranean Challenge,
OpenRobotics is again leveraging simulation to create
virtual environments with three subterranean domains:
tunnels, urban underground, and networks of caves [10].
Sites that would be difficult to access without simulation.
The simulation will allow more teams to compete and
allow a “near infinite set of experiments” bounded only by
construction of test environments and computational
resources [10]. This ability to generate an immense
amount of data and perform a vast number of experiments
is the primary advantage of simulations.
Simulation is also used at a smaller scale for the
Donkey car in the DIY Robocars community. DIY
Robocars is a community started by Chris Anderson in
2017 to encourage building low-cost mobile platforms and
testing autonomous driving algorithms [11]. DIY
Robocars equips remote-control cars with a camera,
controller, and software. By using 1/10 and 1/16 scale
remote-control cars and low-cost hardware, DIY Robocars
helps make self-driving cars accessible to many more
potential developers.
Donkey is an end-to-end neural network library used to
operate some DIY Robocars platforms [4]. Donkey uses
training data consisting of a collection of images and
steering angles collected as a human operator remotely
drives the car around a track. Keras is used to train a
model that given an input image will predict a steering
and throttle output. The steering and throttle are used to
drive the vehicle.
A Unity-based simulator is available to provide
developers with virtual environments for training and
testing of the ML tools [12]. Using the simulator, someone
interested in Donkey can set up and experiment with the
system with no car and no track. The Donkey simulator
uses the Unity 3D game development platform’s graphics
and physics systems to simulate camera input and vehicle
behavior in simple virtual environments. The Donkey
simulator provides three types of environments: a
randomly generated road environment, a recreation of the
DIY Robocars warehouse, and a recreation of a track from
Sparkfun AVC 2017 [13]. There is no expectation that
training will transfer from the simulator to the real-world.
However, with multiple tracks in the simulator, including
102
randomly generated tracks, a user can experiment with
transfer between virtual tracks and could try transferring
training to a real world track.
NVIDIA has developed an autonomous vehicle
simulator that will allow developers to test and validate on
“billions” of simulated miles. NVIDIA’s simulator will
provide “photorealistic” simulation for multiple sensors
(camera, LIDAR, and radar) and will support hardware-in-
the-loop testing [14]. NVIDIA’s simulator attempts to
provide realistic inputs to a complete autonomous vehicle
software stack through a physical interface between a
simulator server and a vehicle server. In comparison with
the Donkey simulator, the NVIDIA system will generate
higher quality images and NVIDIA is providing much of
the content creation. The NVIDIA simulations specifically
provide tools for testing in conditions (adverse weather,
time of day) that would be challenging to test in the real
world and will present difficulties for intelligent systems.
OpenRobotics has developed a vehicle and city
simulation using the Gazebo simulator and the ROS
framework [15]. ROS is a popular framework for
developing robotic systems [16]. The Gazebo simulator
provides a virtual environment that ROS robots can
observe through simulated sensors and perform actions
based on commands from ROS. The OpenRobotics
simulator provides LIDAR, sonar, and camera sensors and
accepts throttle, brake, steer, and gear inputs [15]. Road
network definitions and city models help address a portion
of the content creation challenge.
Video games can also provide a compelling platform
for simulation, particularly for urban environments. The
video game industry spends far more money on creation
of realistic environments for games. Games like GTA V
include a simulated living city complete with a variety of
locations, cars, cyclists, and pedestrians. While they do
not provide a perfect representation of the real world, they
provide a far more realistic one than most research
simulations. However, video games are typically not built
with annotation, data collection, or controls needed for
effective generation of datasets or simulations for testing.
The VIPER dataset is a large annotated synthetic
dataset generated from the realistic environments built in
GTA V [17]. Realistic outdoor environments, particularly
urban environments, require significant content creation to
approach the composition of real world environments.
Richter et al leveraged a video game (GTA V) to generate
a dataset containing over 250,000 high resolution images
with associated annotations and ground truth data for
pixel-level semantic segmentation by instance, detection
and classification of objects, tracking of objects, 3D scene
organization, and visual odometry [17]. In addition, the
resulting image dataset was demonstrated to be similar in
composition and in realism by comparison to other
datasets made up of annotated real-world imagery.
III. ADVANTAGES AND LIMITATIONS
A. Advantages
Accessibility. The cost of robotic hardware is often
prohibitive. In order to work with physical robotic
systems, a developer must be either well-funded or must
scale back to platforms with reasonable cost (e.g., [11]).
In a well-funded environment, it may be impractical to
make available working hardware to entire development
teams. Even in the case of DIY Robocars and the Donkey
car [4, 11], the user must maintain a working vehicle and
have access to one or, ideally, more tracks. Generally, the
only requirement for simulation is that you have access to
computational resources. While more direct and
immediate access is best, the computational resources can
be local or cloud-based resources. In many cases, the
simulation software itself is available at no cost (e.g., [7,
12, 16, 18]).
Hardware Not Required. The primary objective of
simulation is to move research and development to virtual
platforms. By removing the requirement for a physical
prototype, simulation can directly reduce build costs,
design change costs, and time waiting for prototypes.
Physical prototypes have engineering challenges that can
leave development teams waiting on other systems [19].
Infinite Virtual Resources. Real world testing and
training requires real resources. Testing may damage,
destroy, or consume resources purposefully (crash tests,
welding parts) or incidentally (driving into a barrier).
Simulation avoids the cost associated with real resources.
This not only reduces the real cost of testing and training
but also reduces the risk associated with tests. Reduced
cost expands the possibilities for comprehensive testing
including at the edge of performance or of unlikely ideas
[19].
Limited Human Involvement. Simulations can be run
automatically. For software development and unit testing,
simulated tests can be incorporated into individual- or
project-level builds to immediately identify broken
models and codes [8]. A real-world test, especially for an
autonomous vehicle, requires a test site, an operator, and,
for formal testing, a team of individuals to set up scenarios
and collect data. Annotation of data that requires
thousands of person-hours can be generated in real-time
[17, 19]. Removing humans from processes also has the
potential benefit of reducing (not eliminating) biases.
Ease of Experimentation. Simulation provides tools that
make it easier to try out scenarios, modeling methods, and
training strategies. The reduced cost in time and resources
can limit the risk of embarrassing failure and open up the
possibility of testing innovative ideas or running
experiments just to better understand the systems being
built and tested [19].
Ease Data Collection. Simulators like the Donkey
simulator support experimentation for both testing and for
learning. Simulation also helps to ease data collection.
Data collection in the real world requires instrumentation
that can be costly. In some cases, direct measurement is
simply not possible. In simulation, measurements can be
automated and are constrained only by access to the
simulated systems.
Dangerous Conditions. In simulation, we can test in
conditions that would be impossible, impractical, or
unsafe. This includes extreme environments (space,
burning buildings) or simply dangerous conditions
(pedestrian detection and AEB systems).
Increase Engagement. Intelligent systems, particular
embodied systems like robots, are exciting to students.
Through the combined benefits of low cost, accessibility,
reduced risk, and abstraction, simulation allows students
to work on interesting problems.
Variation. Simulation allows near infinite variation,
constrained by computational resources and our ability to
103
build environments [15]. It can be easy in simulation to
modify lighting [14, 17], environments [12, 14, 17, 19],
hardware [19], sensor parameters, algorithms, random
stimuli [1, 12], and visuals [1, 20-25]. There are practical
limitations to real world testing [1]. Real world datasets
and testing expose systems to the complex and
unpredictable variation of the real world but only to the
variation available within those datasets or at those testing
sites. Simulation provides the potential for generating
sufficient variety for effective training while reducing the
effort required for real world training and testing.
Parallelization. Simulations can be distributed across
any number of computer systems. Using simulation, we
can simulate hundreds of thousands of hours of training
[26] or billions of miles of driving [14].
Automated Annotation. Machine learning in particular
requires large annotated datasets. In simulation, we can
automatically generate annotated data sets. The Cityscapes
dataset [3] is a real-world annotated dataset with 5000
frames with pixel-level segmentation and labeling. The
Cityscapes annotations reportedly required an estimated
90 minutes per image and include only 2D segmentation
and labeling. The VIPER dataset is an automatically
generated dataset of 250K frames with comprehensive 2D,
3D, and temporal annotations [17]. To generate a dataset
the size of the VIPER dataset would require
approximately 375,000 person-hours and would only
include a small subset of the annotation data provided by
VIPER.
Abstraction. Abstraction can provide immense benefits
for developing and testing systems in simulation but
abstraction is also the source of the limitations of
simulation. In real-world testing, we must build the
vehicle with a fully implemented powertrain, control
system, drive-by-wire system, computer processors,
sensors, wiring, and more. In simulated testing, by giving
up some fidelity and generalizability, we can abstract
away the details of some systems and other systems
entirely.
B. Limitations
Content Creation. It is possible that content creation is
the most significant barrier to use of simulation for
developing, training, and testing generalizable intelligent
systems, especially if we include the models of physical
systems in our definition of content. Content creation,
solely in terms of models of visual representations of
environments, people, objects, is prohibitively costly in
both time and money. Random generation of objects and
environments have been used successfully, especially
when combined with domain adaptation [4, 26]. We still
require accurate mechanical, electrical, sensor,
communication, and physics models to simulate simple
tasks accurately.
While there are simulation tools and platforms that help
address some aspects of the content creation challenge, no
simulation platform addresses all requirements. VIPER
leverages a video game for excellent environment models,
but experimenters have limited control over the test
environment. Gazebo and Unity-based tools offer
excellent control but limited environments for testing.
Flaws and Limitations. Simulations will never perfectly
match the real world. The benefits of the abstraction
provided by simulation inherently comes with inaccuracy
and, ultimately, challenges generalization and transfer of
learning to real-world applications. Intelligent systems
will identify features that are inherent to the simulation
and will not transfer to the real world [26].
Transfer Simulation to Real. The key challenge is
transfer of intelligent systems from simulation to real
world in ways that allow us to leverage the advantages of
simulation but maintain the generalizability of our models.
Multiple ML methods have demonstrated success in
adapting synthetic semantically-labeled data to realistic
data while retaining accuracy of annotating for training
with results that are equivalent or better than training on
annotated real-world datasets [20-25].
IV. APPLICATIONS
At Mississippi State University, we are currently
developing an autonomous vehicle simulator (MAVS)
[27]. MAVS provides advanced physics-based sensor
simulation and integrates with other simulation products
(ROS, Matlab, Chrono). Internally, we use MAVS to
support the Center for Advanced Vehicular Systems
HALO project [27].
As part of this support, we recently implemented a draft
automated annotation system for two reasons: First, we
wanted to provide a suitable dataset for training and
testing ML algorithms for our specific needs, for which
we do not have an annotated real world dataset nor the
resources to acquire one. Second, we wanted to abstract
part of the sensor-processing task to allow other
developers to work further along the pipeline.
MAVS can now provide automated 2D pixel-level
image segmentation and box annotations for simulated
cameras. Fig. 1 shows a single frame taken from our
simulated camera with the same frame with automated
pixel-level segmentation and box annotation. An initial
test dataset of 10,000 images and annotations was
generated for training and testing.
In addition to the ability to annotate images, MAVS can
also provide all of the other benefits of simulation
previously described. MAVS makes available a simulated
self-driving vehicle platform to any member of our team
willing to learn to operate it. The sensor processing team
is less inconvenienced by necessary mechanical work on
the vehicle. They can generate test data without using fuel,
filling hard drives. MAVS can be used to generate random
paths and automate driving along the paths. MAVS will
allow users to experiment with the vehicle and algorithms
in ways that would not be permissible with the real
vehicle. MAVS allows team members to simultaneously
run simulations without restricting others access to the
vehicle and allows them to run many more experiments
than would be possible in the real world.
MAVS is also challenged by the limitations previously
described. Content creation, especially content relevant to
our particular research interests, has proven to be a
significant challenge. MAVS incorporates some basic
tools to generate random environments but those tools still
use a limited set of visual representations. While MAVS
leverages physics-based models it has clear limitations
with respect to visual fidelity, physical fidelity, and
currently includes no network or communication models.
MAVS is not currently employing domain adaptation
for adapting synthetic data for training for real world
application. Future work for the project will include
104
Figure 1. Simulated camera image (left) and segmented and box-
annotated image (right).
evaluating the available methods for domain adaptation
and applying them to our synthetic datasets.
V. SUMMARY
Intelligent systems require large datasets for training
and testing. Simulation provides compelling advantages
for collecting datasets that include comprehensive
automated annotations for training and testing. In addition,
simulation provides additional advantages for
development and testing of intelligent systems. However,
simulation has limited generalizability and faces
significant challenges in construction of virtual
environments and underlying modeling and simulation
codes. Current research is exploring methods to address
limitations but simulation will never perfectly match the
physical world. Developers using simulation must be
aware of the limitations of their simulation tools and the
potential impact on generalizability of their models.
ACKNOWLEDGMENT
The author would like to thank C. Goodin for his
assistance and guidance for developing in MAVS and the
improved implementation of the initial effort.
REFERENCES
[1] K. Bousmalis, A. Irpan, P. Wohlhart, Y. Bai, M. Kelcey, M.
Kalakrishnan, L. Downs, J. Ibarz, P. Pastor, K. Konolige, S.
Levine, and V. Vanhoucke, “Using Simulation and Domain
Adaptation to Improve Efficiency of Deep Robotic Grasping,”
IEEE International Conference on Robotics and Automation
(ICRA), Brisbane, 2018.
[2] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z.
Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and L.
Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,”
Int. J. Comput. Vis., vol. 115, pp. 211-252, 2015.
[3] M. Cordts, M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R.
Benenson, U. Franke, S. Roth, and B. Schiele, “The Cityscapes
Datasets for Semantic Urban Scene Understanding,” in Proc. Of
the IEEE Conference on Computer Vision and Pattern
Recognition (CVPR), 2016.
[4] W. Roscoe, “Donkey Car: An opensource DIY self driving
platform for small scale cars,” [Online]. Available:
http://www.donkeycar.com [Accessed: July 10, 2018].
[5] “DARPA Robotics Challenge (DRC),” [Online]. Available:
https://www.darpa.mil/program/darpa-robotics-challenge
[Accessed July 10, 2018].
[6] Boston Dynamics, “Boston Dynamics Atlas Robot.” [Online].
Available: http://www.bostondynamics.com/robot_Atlas.html
[Accessed: July 12, 2018].
[7] C.E. Aguero, N. Koenig, I. Chen, H. Boyer, S. Peters, J. Hsu, B.
Gerkey, S. Paepcke, J.L. Rivero, J. Manzo, E. Krotkov, and G.
Pratt, “Inside the Virtual Robotics Challenge: Simulating Real-
Time Robotic Disaster Response,” IEEE Transactions on
Automation Science and Engineering, vol. 12 (2), pp. 494-506,
April 2015.
[8] M. Johnson, B. Shrewsbury, S. Bertrand, T. Wu, D. Duran, M.
Floyd, P. Abeles, D. Stephen, N. Mertins, A. Lesman, J. Carff, W.
Rifenburgh, P. Kaveti, W. Straatman, J. Smith, M. Griffioen, B.
 Layton, T. de Boer, T. Koolen, P. Neuhaus, and J. Pratt, “Team [19] D.W. Carruth, “Virtual Reality for Education and Workforce
IHMC’s Lessons Learned from the DARPA Robotics Challenge Training,” in Proceedings of 2017 15th International Conference
Trials,” Journal of Field Robotics, vol. 32, no. 2, pp. 192-208, on Emerging eLearning Technologies and Applications (ICETA),
2015. 2017, pp. 1-6. doi: 10.1109/ICETA.2017.8102472.
[9] L. Jensen, “Evaluating the Validity of Latency Effects in Robotics [20] A. Shrivastava, T. Pfister, O. Tuzel, J. Susskind, W. Wang, and R.
Simulation,” M.S. thesis, Mississippi State University, Starkville, Webb, “Learning from Simulated and Unsupervised Images
MS, 2017. through Adversarial Training,” IEEE Conference on Computer
[10] M. Simon, “DARPA’s Next Challenge? A Grueling Underground Vision and Pattern Recognition (CVPR), Honolulu, HI, 2017, pp.
Journey,” Wired, [Online]. Available: 2242-2251. doi: 10.1109/CVPR.2017.241
https://www.wired.com/story/darpas-next-challenge-a-grueling- [21] Y. Ganin, E. Ustinova, H. Ajakan, P. Germain, H. Larochelle, F.
underground-journey/ [Accessed: July 13, 2018]. Laviolette, M. Marchand, and V. Lempitsky, “Domain-
[11] “DIY Robocars,” [Online]. Available: https://diyrobocars.com/ Adversarial Training of Neural Networks,” Journal of Machine
[Accessed: July 14, 2018]. Learning Research, vol. 17, pp. 1-35, 2016.
[12] “Donkey Simulator,” [Online]. Available: [22] J. Tobin, R. Fong, A. Ray, J. Schneider, W. Zaremba, and P.
http://www.donkeycar.com/updates/simulator [Accessed: July 15, Abbeel, “Domain Randomization for Transferring Deep Neural
2018]. Networks from Simulation to the Real World,” in Proceedings
30th IEEE/RSJ International Conference on Intelligent Robots and
[13] T. Kramer. “Sparkfun AVC 2017 Donkey Simulator,” YouTube,
Systems (IROS), Vancouver, Canada, 2017.
October 19, 2017 [Video File]. Available:
https://youtu.be/_4fDMYh4ICA. [Accessed: July 14, 2018]. [23] E. Tzeng, C. Devin, J. Hoffman, C. Finn, P. Abbeel, S. Levine, K.
Saenko, and T. Darrell, “Adapting Deep Visuomotor
[14] “NVIDIA DRIVE Constellation: Virtual reality autonomous
vehicle simulator,” [Online]. Available: Representations with Weak Pairwise Constraints,” In Workshop
on the Algorithmic Foundations of Robotics (WAFR), 2016.
https://www.nvidia.com/en-us/self-driving-cars/drive-
constellation/ [Accessed: July 15, 2018]. [24] F. Sadeghi and S. Levine, CAD2RL: Real Single-Image Flight
[15] I. Chen and C. Aguero, “Vehicle and city simulation with Gazebo Without a Single Real Image,” in Proceedings of Robotics:
Science and Systems, Cambridge, MA, 2017. doi:
and ROS,” presented at ROSCON 2017, Vancouver, Canada,
2017, [Online]. Available: 10.15607/RSS.2017.XIII.034
https://roscon.ros.org/2017/presentations/ROSCon%202017%20V [25] P. Li, X. Liang, D. Jia, and E. Xing, “Semantic-aware Grad-GAN
ehicle%20and%20City%20Simulation.pdf [Accessed: July 15, for Virtual-to-Real Urban Scene Adaptation,” arXiv:1801.01726
2018]. [cs.CV], Jan. 2018.
[16] M. Quigley, K. Conley, B.P. Gerkey, J. Faust, T. Foote, J. Leibs, [26] K. Bousmalis and S. Levine, “Closing the Simulation-to-Reality
R. Wheeler, and A. Ng, “ROS: an open-source Robot Operating Gap for Deep Robotic Learning,” Google AI Blog, October 30,
System,” 2009 ICRA Workshop on Open Source Software, Kobe, 2017. [Online]. Available:
Japan, 2009. https://ai.googleblog.com/2017/10/closing-simulation-to-reality-
gap-for.html [Accessed: July 14, 2018].
[17] S.R. Richter, Z. Hayder, and V. Koltun, “Playing for
Benchmarks,” 2017 IEEE International Conference on Computer [27] C. Hudson, C. Goodin, M. Doude, and D.W. Carruth, “Analysis of
Vision (ICCV), Venice, 2017, pp. 2232-2241. doi: Dual LIDAR Placement for Off-Road Autonomy Using MAVS,”
10.1109/ICCV.2017.243 World Symposium on Digital Intelligence for Systems and
[18] “Unity Machine Learning,” [Online], Available: Machines, Kosice, Slovakia, 2018.
https://unity3d.com/machine-learning [Accessed: July 14, 2018].

105
106