@vpalmisano/webrtcperf
WebRTC Perf
WebRTC performance and quality evaluation tool. It allows to validate the audio/video quality and the client CPU/memory usage when multiple connections join the same WebRTC service.
Main features:
- A NodeJS application/library using Puppeteer for controlling chromium instances.
- It can be executed:
- using the pre built Docker image; this is the suggested way to run the tool without installing any dependency;
- from sources (using git pull or npm install);
- using the pre built executables generated for each platform.
- It allows to inject custom Javascript source files that will run into the browser page context for automating some tasks (e.g. pressing a button to join a conference room).
- It allows to throttle the networking configuration, limiting the ingress/egress available bandwidth, the RTT or the packet loss %.
- It uses a patched version of chromium (see
./chromiumdirectory) that allows to disable the video decoding, lowering the CPU requirements when running multiple browser sessions. - It contains an RTC stats logging module that allows to collect metrics and send them to a Prometheus Pushgateway server for live visualization with Grafana.
- It allows to override getUserMedia and getDisplayMedia calls.
- It allows to define alert rules and generate reports.
Install
The tool can be executed from sources, using the pre built executables or using the Docker image.
Using Npm:
echo '@vpalmisano:registry=https://npm.pkg.github.com' >> ~/.npmrc
npm install -g @vpalmisano/webrtcperf
# Install FFMpeg:
sudo apt install ffmpeg # Linux
# or:
brew install ffmpeg # MacOS
# Run a Jitsi test:
webrtcperf \
--url="https://meet.jit.si/${JITSI_ROOM_URL}#config.prejoinPageEnabled=false" \
--display='' \
--show-page-log=false
# Press <q> to stop.
Using Docker:
docker pull ghcr.io/vpalmisano/webrtcperf
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url="https://meet.jit.si/$JITSI_ROOM_URL#config.prejoinPageEnabled=false" \
--show-page-log=false \
--sessions=1 \
--tabs-per-session=1
Stop the tool pressing q (normal browser close) or x (it will close the
process immediately).
Configuration options
See the config documentation.
Statistics
Example output:
-- Mon, 06 Feb 2023 20:46:34 GMT -------------------------------------------------------------------
name count sum mean stddev 5p 95p min max
System CPU 1 15.89 0.00 15.89 15.89 15.89 15.89 %
System GPU 1 0.00 0.00 0.00 0.00 0.00 0.00 %
System Memory 1 72.18 0.00 72.18 72.18 72.18 72.18 %
CPU/page 1 84.42 84.42 0.00 84.42 84.42 84.42 84.42 %
Memory/page 1 1206.90 1206.90 0.00 1206.90 1206.90 1206.90 1206.90 MB
Pages 1 1 1 0 1 1 1 1
Errors 1 0 0 0 0 0 0 0
Warnings 1 0 0 0 0 0 0 0
Peer Connections 1 2 2 0 2 2 2 2
-- Inbound audio -----------------------------------------------------------------------------------
rate 2 28.73 14.36 14.36 0.00 28.73 0.00 28.73 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
jitter 2 0.00 0.00 0.00 0.00 0.00 0.00 s
avgJitterBufferDelay 1 35.29 0.00 35.29 35.29 35.29 35.29 ms
-- Inbound video -----------------------------------------------------------------------------------
received 2 2.66 1.33 1.32 0.01 2.64 0.01 2.64 MB
rate 2 967.41 483.71 483.71 0.00 967.41 0.00 967.41 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
jitter 2 0.01 0.01 0.01 0.02 0.01 0.02 s
avgJitterBufferDelay 1 50.48 0.00 50.48 50.48 50.48 50.48 ms
width 2 960 320 640 1280 640 1280 px
height 2 540 180 360 720 360 720 px
fps 1 15 0 15 15 15 15 fps
-- Outbound audio ----------------------------------------------------------------------------------
rate 2 42.84 21.42 21.42 0.00 42.84 0.00 42.84 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
roundTripTime 1 0.001 0.000 0.001 0.001 0.001 0.001 s
-- Outbound video ----------------------------------------------------------------------------------
sent 2 3.25 1.62 1.58 0.04 3.21 0.04 3.21 MB
rate 2 1131.25 565.63 565.63 0.00 1131.25 0.00 1131.25 Kbps
lost 1 0.00 0.00 0.00 0.00 0.00 0.00 %
roundTripTime 1 0.001 0.000 0.001 0.001 0.001 0.001 s
qualityLimitResolutionChanges 2 2 1 1 0 2 0 2
qualityLimitationCpu 2 0 0 0 0 0 0 0 %
qualityLimitationBandwidth 2 20 10 10 0 20 0 20 %
sentActiveEncodings 2 2 1 1 3 1 3 encodings
sentMaxBitrate 2 3700.00 1850.00 350.00 1500.00 2200.00 1500.00 2200.00 Kbps
width 2 640 640 0 1280 0 1280 px
height 2 360 360 0 720 0 720 px
fps 2 12 12 0 25 0 25 fps
pliCountReceived 2 1 0 1 2 1 2
Statistics values:
| Name | Count | Description |
|---|---|---|
| cpu | Total sessions | The browser process cpu usage. |
| memory | Total sessions | The browser process memory usage. |
| tabs | Total sessions | The browser current opened tabs. |
| received | Total inbound streams | The bytesReceived value for each stream. |
| sent | Total outbound streams | The bytesSent value for each stream. |
| retransmitted | Total outbound streams | The retransmittedBytesSent value for each stream. |
| rate | Total streams | The stream bitrate. |
| lost | Total streams | The stream lost packets %. |
| jitter | Total streams | The stream jitter in seconds. |
| avgJitterBufferDelay | Total decoded tracks | The inbound average jitter buffer delay. |
| qualityLimitResolutionChanges | Total outbound video streams | The qualityLimitationResolutionChanges value for each outbound video stream. |
| width | Total sent or received videos | The sent or received video width. |
| height | Total sent or received videos | The sent or received video height. |
| fps | Total sent | The sent video frames per second. |
Prometheus / Grafana
See the prometheus stack.
Examples
Mediasoup demo
Starts one send-receive participant:
docker run -it --rm --name=webrtcperf-publisher \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$MEDIASOUP_DEMO_URL \
--url-query='roomId=test&displayName=Publisher($s-$t)' \
--sessions=1 \
--tabs-per-session=1
Starts 10 receive-only participants:
docker run -it --rm --name=webrtcperf-viewer \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$MEDIASOUP_DEMO_URL \
--url-query='roomId=test&displayName=Viewer($s-$t)&produce=false' \
--sessions=1 \
--tabs-per-session=10
Edumeet
Starts one send-receive participant, with a random audio activation pattern:
docker run -it --rm \
-v /dev/shm:/dev/shm \
-v $PWD/examples:/scripts:ro \
ghcr.io/vpalmisano/webrtcperf \
--url=$EDUMEET_URL \
--url-query='displayName=Publisher($s-$t)' \
--script-path=/scripts/edumeet-sendrecv.js \
--sessions=1 \
--tabs-per-session=1
Starts 10 receive-only participants:
docker run -it --rm \
-v /dev/shm:/dev/shm \
-v $PWD/examples:/scripts:ro \
ghcr.io/vpalmisano/webrtcperf \
--url=$EDUMEET_URL \
--url-query='displayName=Viewer($s-$t)' \
--script-path=/scripts/edumeet-recv.js \
--sessions=1 \
--tabs-per-session=10
Jitsi
Starts one send-receive participant:
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$JITSI_ROOM_URL \
--url-query='#config.prejoinPageEnabled=false&userInfo.displayName=Participant($s-$t)' \
--sessions=1 \
--tabs-per-session=1
Starts 10 receive-only participants:
docker run -it --rm \
-v /dev/shm:/dev/shm \
ghcr.io/vpalmisano/webrtcperf \
--url=$ROOM_URL \
--url-query='#config.prejoinPageEnabled=false&userInfo.displayName=Participant($s-$t)' \
--sessions=1 \
--tabs-per-session=10
Running from source code
The DEBUG_LEVEL environment variable can be used to enable debug messages;
see debug-level for syntax.
git clone https://github.com/vpalmisano/webrtcperf.git
cd webrtcperf
# Optional: build the chromium customized version
# cd chromium
# ./build.sh setup
# ./build.sh apply_patch
# ./build.sh build
# install the package (on Ubuntu/Debian)
# dpkg -i ./chromium-browser-unstable_<version>-1_amd64.deb
# cd ..
yarn build
# sendrecv test
DEBUG_LEVEL=DEBUG:* yarn start \
--url=https://127.0.0.1:3443/test \
--url-query='displayName=SendRecv($s/$S-$t/$T)' \
--script-path=./examples/edumeet-sendrecv.js \
--sessions=1 \
--tabs-per-session=1
# recv only
DEBUG_LEVEL=DEBUG:* yarn start \
--url=https://127.0.0.1:3443/test \
--url-query='displayName=Recv($s/$S-$t/$T)' \
--script-path=./examples/edumeet-recv.js \
--sessions=1 \
--tabs-per-session=10
Using the VMAF calculator
- Run the tool adding the following options:
--script-params="{timestampWatermarkVideo:true,saveRecvVideoTrack:1}"
--server-port=5000
--server-use-https=true
--server-data=/data
With saveRecvVideoTrack you can specify the sessions that will be saved at
receiver side (in this example 1 will save all the streams received in the
sessions with index 0 and 1).
2. The sent/received videos will be saved in the /data directory.
3. Run the VMAF calculator comparing the sent/received videos:
docker run --rm \
-e DEBUG_LEVEL=INFO \
-v $PWD/data:/data \
ghcr.io/vpalmisano/webrtcperf:devel \
--vmaf-path /data \
The tool will generate a .vmaf.json and a .plotly files in the data/vmaf directory.
Adding the --vmaf-preview option, a .mp4 file containing the side-by-side
video comparison will be generated.
Authors
- Vittorio Palmisano [github]