This section describes some relatively simple benchmarks performed
to compare the execution time of simple builds under the Arduino
IDE with those done by dno. In each case builds were of a
slightly modified blink.ino file as shown
below.
The tests were run on a relatively old x86_64 GNU/Linux desktop machine between dno version 0.9.3 and the Arduino IDE version 1.8.13.
Each benchmark was performed by running the benchmark 5 times and discarding the first result. This allows the OS cache to be initialised as it would likely be in a normal running situation.
We timed 2 distinct invocations of IDE builds: one for an Arduino Pro, and one for an esp32 (coming soon).
Since automating the timing of an IDE build is difficult, we ran initial builds in the IDE, and then copied and pasted the commands run by the IDE, into a script. We then performed timed runs of that script.
For all tests, output is redirected to /dev/null in order that processing the terminal output should have as little impact on timing as possible.
Our blink.ino file looked like this:
/*
Blink
Turns on an LED on for one second, then off for one second, repeatedly.
Most Arduinos have an on-board LED you can control. On the UNO, MEGA and ZERO
it is attached to digital pin 13, on MKR1000 on pin 6. LED_BUILTIN is set to
the correct LED pin independent of which board is used.
If you want to know what pin the on-board LED is connected to on your Arduino model, check
the Technical Specs of your board at https://www.arduino.cc/en/Main/Products
This example code is in the public domain.
modified 8 May 2014
by Scott Fitzgerald
modified 2 Sep 2016
by Arturo Guadalupi
modified 8 Sep 2016
by Colby Newman
*/
#ifndef LED_BUILTIN
#define LED_BUILTIN 13
#endif
// the setup function runs once when you press reset or power the board
void setup() {
// initialize digital pin LED_BUILTIN as an output.
pinMode(LED_BUILTIN, OUTPUT);
}
// the loop function runs over and over again forever
void loop() {
digitalWrite(LED_BUILTIN, HIGH); // turn the LED on (HIGH is the voltage level)
delay(1000); // wait for a second
digitalWrite(LED_BUILTIN, LOW); // turn the LED off by making the voltage LOW
delay(1000); // wait for a second
}
This set of comparisons is for a "Full Build". That is to say, a build where all sources are compiled from scratch.
Our script for this was as follows:
#! /usr/bin/env bash
rm /tmp/arduino_cache_248731/core/core_arduino_avr_pro_cpu_16MHzatmega328_1621df717313d057c92202babd71649a.a
find /tmp/arduino_build_271265/ -name '*.o' | xargs rm
do_it()
{
arduino-builder -dump-prefs -logger=machine -hardware /usr/share/arduino/hardware -tools /usr/share/arduino/hardware/tools/avr -libraries /home/marc/Arduino/libraries -fqbn=arduino:avr:pro:cpu=16MHzatmega328 -ide-version=10813 -build-path /tmp/arduino_build_271265 -warnings=none -build-cache /tmp/arduino_cache_248731 -prefs=build.warn_data_percentage=75 -verbose /home/marc/Arduino/sketch_may10b/sketch_may10b.ino
arduino-builder -compile -logger=machine -hardware /usr/share/arduino/hardware -tools /usr/share/arduino/hardware/tools/avr -libraries /home/marc/Arduino/libraries -fqbn=arduino:avr:pro:cpu=16MHzatmega328 -ide-version=10813 -build-path /tmp/arduino_build_271265 -warnings=none -build-cache /tmp/arduino_cache_248731 -prefs=build.warn_data_percentage=75 -verbose /home/marc/Arduino/sketch_may10b/sketch_may10b.ino
}
time do_it
This script removes the cached libcore.a file and all
previously compiled files, before executing the same commands
that the Arduino IDE would issue using the
time command to capture timing information.
Here are the commands which were run and the results:
blink$ ./cli.sh >/dev/null real 0m2.798s user 0m1.432s sys 0m1.560s blink$ ./cli.sh >/dev/null real 0m2.569s user 0m1.250s sys 0m1.502s blink$ ./cli.sh >/dev/null real 0m2.621s user 0m1.308s sys 0m1.481s blink$ ./cli.sh >/dev/null real 0m2.484s user 0m1.299s sys 0m1.363s blink$
We created a simple dno project tree as follows:
blink ├── blink.ino └── pro.8MHzatmega328
In this set of builds, we rebuild everything including the
BOARD_INFO file which normally does not
need to be touched. This is the dno build that is most
similar to the IDE build.
marc:pro.8MHzatmega328$ dno pristine; time dno >/dev/null Super-cleaning . real 0m1.911s user 0m1.805s sys 0m0.413s pro.8MHzatmega328$ dno pristine; time dno >/dev/null Super-cleaning . real 0m1.919s user 0m1.813s sys 0m0.431s pro.8MHzatmega328$ dno pristine; time dno >/dev/null Super-cleaning . real 0m1.881s user 0m1.784s sys 0m0.418s pro.8MHzatmega328$ dno pristine; time dno >/dev/null Super-cleaning . real 0m1.838s user 0m1.763s sys 0m0.395s pro.8MHzatmega328$
In this set of builds, we rebuild everything except the
BOARD_INFO file. This is what a normal
clean build would look like.
pro.8MHzatmega328$ dno clean; time dno >/dev/null Cleaning . real 0m1.441s user 0m1.295s sys 0m0.307s pro.8MHzatmega328$ dno clean; time dno >/dev/null Cleaning . real 0m1.476s user 0m1.308s sys 0m0.328s pro.8MHzatmega328$ dno clean; time dno >/dev/null Cleaning . real 0m1.475s user 0m1.316s sys 0m0.321s pro.8MHzatmega328$ dno clean; time dno >/dev/null Cleaning . real 0m1.475s user 0m1.326s sys 0m0.310s pro.8MHzatmega328$
For this set of builds we pass the flags -j
4 to dno. This causes it to run as many as 4
commands in parallel when it can. This is expected to reduce
the elapsed build time.
This would be the natural, and fastest way to run dno.
pro.8MHzatmega328$ dno clean; time dno -j 4 >/dev/null Cleaning . real 0m0.818s user 0m1.273s sys 0m0.306s pro.8MHzatmega328$ dno clean; time dno -j 4 >/dev/null Cleaning . real 0m0.820s user 0m1.257s sys 0m0.321s pro.8MHzatmega328$ dno clean; time dno -j 4 >/dev/null Cleaning . real 0m0.816s user 0m1.288s sys 0m0.286s pro.8MHzatmega328$ dno clean; time dno -j 4 >/dev/null Cleaning . real 0m0.819s user 0m1.299s sys 0m0.279s pro.8MHzatmega328$
In this scenario, we do a more typical compilation operation, where the core library does not need to be rebuilt.
For this scenario, we re-used our
cli.shoriginal script with the
rm commands commented out.
blink$ ./cli.sh >/dev/null real 0m1.842s user 0m0.793s sys 0m1.212s blink$ ./cli.sh >/dev/null real 0m1.828s user 0m0.732s sys 0m1.260s blink$ ./cli.sh >/dev/null real 0m1.791s user 0m0.678s sys 0m1.263s blink$ ./cli.sh >/dev/null real 0m1.741s user 0m0.789s sys 0m1.102s blink$
For this scenario, we touch the
blink.ino file so that it appears to have
been updated. We then run dno and see the following.
pro.8MHzatmega328$ touch ../blink.ino; time dno >/dev/null real 0m0.528s user 0m0.541s sys 0m0.109s pro.8MHzatmega328$ touch ../blink.ino; time dno >/dev/null real 0m0.523s user 0m0.542s sys 0m0.104s pro.8MHzatmega328$ touch ../blink.ino; time dno >/dev/null real 0m0.520s user 0m0.538s sys 0m0.107s pro.8MHzatmega328$ touch ../blink.ino; time dno >/dev/null real 0m0.526s user 0m0.546s sys 0m0.105s pro.8MHzatmega328$
Note that the output above shows the first, ignored, dno run
where we have not discarded stdout. This
is shown simply to show that the dno build did indeed do
something. Otherwise, given the speed of the build, you could
be forgiven for thinking that this test was somehow a cheat.
This is the same as above, except that we run dno with no
updated sources. This is equivalent to the situation with the
cli.sh script above, but dno is smart
enough to perform no build.
pro.8MHzatmega328$ time dno >/dev/null real 0m0.341s user 0m0.378s sys 0m0.059s pro.8MHzatmega328$ time dno >/dev/null real 0m0.342s user 0m0.373s sys 0m0.066s pro.8MHzatmega328$ time dno >/dev/null real 0m0.339s user 0m0.362s sys 0m0.073s pro.8MHzatmega328$ time dno >/dev/null real 0m0.340s user 0m0.386s sys 0m0.049s pro.8MHzatmega328$
Table 9.1. Full Build for Arduino Pro: performance comparison
| Cumulative Seconds (4 runs) | Comparative Time (% of baseline) | |||||||
|---|---|---|---|---|---|---|---|---|
| Real | User | Sys | U+S | Real % | User % | Sys % | U+S % | |
| Arduino IDE (baseline 1) | 10.472 | 5.289 | 5.906 | 11.195 | ||||
| dno, rebuilding BOARD_INFO (against baseline 1) | 7.549 | 7.165 | 1.657 | 8.822 | 72.1 | 135.5 | 28.1 | 78.8 |
| dno, preserved BOARD_INFO (against baseline 1) | 5.867 | 5.245 | 1.266 | 6.511 | 56.0 | 99.2 | 21.4 | 58.2 |
| parallel dno (-j 4) (against baseline 1) | 3.273 | 5.117 | 1.192 | 6.309 | 31.3 | 96.7 | 20.2 | 56.4 |
| Arduino IDE minimal build (baseline 2) | 7.202 | 2.992 | 4.837 | 7.829 | ||||
| dno, rebuilding blink.ino (against baseline 2) | 2.097 | 2.167 | 0.425 | 2.592 | 29.1 | 72.4 | 8.8 | 33.1 |
| dno, no updates (against baseline 2) | 1.362 | 1.499 | 0.247 | 1.746 | 18.9 | 50.1 | 5.1 | 22.3 |