A Novel
Fatigue Driving State Recognition and Warning
Method Based on EEG and EOG
Signals
Li Liu, Yunfeng Ji, Yun Gao, Zhenyu Ping,
Liang Kuang , Tao Li, and Wei Xu
Abstract
Traffic accidents are easily caused by tired
driving. If the fatigue state of the driver can
be identified in time and a corresponding early
warning can be provided, then the occurrence of
traffic accidents could be avoided to a large
extent. At present, the recognition of fatigue
driving states is mostly based on recognition
accuracy. Fatigue state is currently recognized
by combining different features, such as facial
expressions, electroencephalogram (EEG) signals,
yawning, and the percentage of eyelid closure
over the pupil over time (PERCLoS). The
combination of these features increases the
recognition time and lacks real-time
performance. In addition, some features will
increase error in the recognition result, such
as yawning frequently with the onset of a cold
or frequent blinking with dry eyes. On the
premise of ensuring the recognition accuracy and
improving the realistic feasibility and
real-time recognition performance of fatigue
driving states, a fast support vector machine
(FSVM) algorithm based on EEGs and
electrooculograms (EOGs) is proposed to
recognize fatigue driving states. First, the
collected EEG and EOG modal data are
preprocessed. Second, multiple features are
extracted from the preprocessed EEGs and EOGs.
Finally, FSVM is used to classify and recognize
the data features to obtain the recognition
result of the fatigue state. Based on the
recognition results, this paper designs a
fatigue driving early warning system based on
Internet of Things (IoT) technology. When the
driver shows symptoms of fatigue, the system not
only sends a warning signal to the driver but
also informs other nearby vehicles using this
system through IoT technology and manages the
operation background.
Résumé
Les accidents de la route sont souvent
causés par la somnolence au volant. Si
l'état de fatigue du conducteur peut
être identifié à temps et
qu'une alerte précoce peut être
fournie, alors l'occurrence des accidents de la
route pourrait être réduite dans
une large mesure. L'état de fatigue est
actuellement reconnu en combinant
différentes caractéristiques,
telles que les expressions faciales, les signaux
d'électroencéphalogramme (EEG), le
bâillement et le pourcentage de fermeture
des paupières sur la pupille au fil du
temps (PERCLoS).
La combinaison de ces paramètres
allonge le délai de la reconnaissance et
manque de performances en temps réel. De
plus, certaines données augmentent les
erreurs de détection, comme le
bâillement fréquent au début
d'un rhume ou des clignements fréquents
liés à des yeux secs. Dans le but
d'améliorer la précision de la
détection et d'améliorer une
faisabilité réaliste en temps
réel de la fatigu et la somnolence, un
algorithme (FSVM) basé sur des EEG et des
électro-oculogrammes (EOG) est
proposé pour reconnaître les
états de somnolence au volant. Tout
d'abord, les données EEG et EOG
collectées sont
prétraitées. Puis, plusieurs
caractéristiques sont extraites des EEG
et EOG ainsi prétraités. FSVM est
utilisé pour classer et reconnaître
les caractéristiques des données
nécessaires à la reconnaissance de
la somnolence. Cet article rapporte les
résultats apportés par un
système d'alerte précoce en
matière de somnolence basé sur la
technologie Internet des objets (IoT). Lorsque
le conducteur présente des
symptômes, le système envoie non
seulement un signal d'avertissement au
conducteur, mais informe également les
autres véhicules à
proximité utilisant ce système via
la technologie IoT et gère
l'arrière-plan de l'opération.
1. Introduction
Fatigue is a very complex physical and
psychological state that can be divided into
mental fatigue and physical fatigue. In most
cases, mental fatigue and physical fatigue are
intertwined and appear at the same time. Mental
fatigue is often caused by long-term cognitive
activity in the brain. Under brain fatigue,
people's cognitive function is limited, and
their alertness is reduced. Drivers are prone to
both mental and physical fatigue during
long-term driving, but mental fatigue is the
main problem. Fatigue driving is one of the
major hidden dangers of road traffic safety.
Research on fatigue driving recognition and
early warning technology can reduce the
frequency of traffic accidents [1].
Fatigue driving status recognition is a
prerequisite for early warning, so fatigue
driving status recognition is very important. At
present, research on fatigue driving
identification methods mainly focuses on three
aspects: (1) identification based on driver
behavior characteristics: the driver's fatigue
state is judged by the recognition of the
driver's behavior, such as the movement of the
eyelids, the closed state of the eyes
[2], and facial expressions [3].
The identification method is simple and easy to
implement, but the scoring standard is easily
affected by conditions such as personal
behavior, light, and image acquisition angle.
The collection of various modal data will
inevitably be noisy, causing the recognition
result to fail to correctly identify the
driver's fatigue state. (2) Detection based on
vehicle parameters: through the detection of
vehicle parameters such as vehicle speed,
vehicle position, and steering wheel rotation
angle during driving, the driver's operating
indicators are obtained, and then the degree of
fatigue is judged. Since vehicle parameters are
closely related to the actual driving quality of
the driver, this method is closer to the actual
driving situation. However, vehicle parameters
need to be measured during actual operation,
which increases the cost of the vehicle. (3)
Recognition based on the physiological
parameters of the driver: the driver's fatigue
state can be judged by identifying the driver's
physiological characteristics, such as with
electrocardiograms [4],
electroencephalograms [5, 6],
electrooculograms [7], and
electromyography [8, 9].
Since the EEG signal directly reflects the
driver's brain activity and the price of EEG
signal acquisition devices are declining, they
are therefore convenient to use. Therefore,
identifying driving fatigue states based on the
EEG signals is considered to be one of the most
objective and accurate analysis methods.
Reference [5] proposed a new real-time
fatigue driving detection method based on EEG
signals. The study combines two characteristics
of power spectral density (PSD) and sample
entropy (SampEn) to judge mental fatigue. The
results show that the method is effective for
fatigue detection because the prediction results
of fatigue are consistent with the phenomena
recorded in the simulated driving process. This
is considered an objective measure of behavior.
Reference [10] proposed a recurrent
network-based convolutional neural network
(RN-CNN) method to detect fatigue driving. The
data used in the experiment are the EEG signals
collected during driving simulation. This method
can achieve an average recognition accuracy of
92.95%. Reference [11] proposed the
detection of the fatigue driving state based on
the feature data of sample entropy, approximate
entropy, and complexity, which can well identify
four different mental fatigue states. Reference
[12] uses five different entropies, the
relative energy of the alpha wave, and as
indicators for judging fatigue. The experimental
results show that this fusion method can
accurately judge the fatigue degree of the
driver. Reference [13] uses the fast
Fourier transform to extract four rhythm
features. By analyzing the trend and mutual
relationship of these four features, it is found
that using as the feature to assess the mental
state is the most effective. Reference
[14] studied sample entropy, fuzzy
entropy, approximate entropy, and spectral
entropy as the inputs of a decision tree.
Experiments have shown that this method has an
accuracy of 94% for the identification of
fatigue driving, and it can identify fatigue
driving more accurately. Typical EEG signal
characteristic analysis methods are mainly
divided into the time domain [15],
frequency domain [16], and
time-frequency domain analysis methods
[17]. The EEG signal in the frequency
domain has obvious characteristics and strong
distinguishability. It is of great significance
to the analysis of EEG signals. EEG signals in
different frequency bands can effectively
reflect people's mental state and excitement
[18]. Reference [19] uses a
convolutional neural network (CNN) to realize
emotion recognition based on the time-frequency
diagram of EEG signals obtained by a wavelet
transform. However, the time-frequency diagram
cannot effectively reflect the correlation of
EEG signals between different electrodes. The
above studies have shown that fatigue driving
recognition based on EEGs is the most objective
and accurate fatigue recognition method and is
known as the "gold standard" of fatigue
detection.
Fatigue driving state recognition based on
EOGs mainly separates the horizontal and
vertical EOG signals from the electrode signals
of the forehead and extracts a series of
features, such as gaze, blinking, and saccade,
for driver fatigue state recognition. Reference
[20] found that as driver fatigue
increases, it will be accompanied by long-term
blinking, which reflects the relationship
between slow eye movement and driver fatigue. By
extracting the eye movement features in the EOG
signal, machine learning algorithms are used to
identify the driver's fatigue state. Reference
[21] detected fatigue driving by
extracting the fatigue characteristics of
blinking, slow eye movement, amplitude, and
periodicity in the EOG signal, and the
experimental results showed that the detection
effect was effective. In summary, driver fatigue
detection based on EOG signal characteristics is
also feasible.
At present, most studies mainly focus on the
fusion of multiple features and the application
of integrated classifiers. The purpose of these
studies is to maximize the accuracy of fatigue
recognition. However, most studies ignore the
real-time performance of fatigue driving
recognition and early warning. In a real-life
environment, the timely identification and early
warning of fatigue driving are more meaningful.
With the rapid development of modern industry,
collection methods of EEG and EOG signals are
becoming more advanced. The volume of collection
equipment is becoming increasingly miniaturized
and portable, their collection accuracy is
increasing, and their production cost is
decreasing. With the development of Internet of
Things (IoT) technology in recent years, it is
no longer difficult to collect driver EEG
signals without interference. Based on the above
background, this paper proposes a fast
identification method of fatigue driving based
on EEG and EOG. This method can collect the
driver's EEG and EOG signals in a real
environment and complete rapid identification
and timely warning. The contents of this study
can be divided as follows:
(1) More objective EEG and EOG signal data
are used as the identification data of the
fatigue driving state. For EEG data, its PSD and
differential entropy are extracted as feature
data. For EOG, EOG features extracted based on
independent component analysis
(features_table_ica), EOG features extracted
based on subtraction rules
(features_table_minus), and EOG features
extracted using both subtraction rules and
principal component analysis
(features_table_icav_minh) are used as feature
data. The multifeature data of the two
modalities can represent more comprehensive
sample information.
(2) A fast SVM algorithm based on sample
geometric features is proposed. For the case of
nonlinear separability, the support vector in
the high-dimensional space should also be on the
edge of the positive and negative classes.
Measured by distance, the support vector is
composed of those sample points with larger
distances of the same kind and smaller distances
of different kinds. The key is to find such
sample points. FSVM can greatly reduce the
number of training samples and reduce the number
of support vectors, resulting in a reduction in
the training time of the model, and at the same
time, the impact on sample classification
accuracy is minimal.
(3) Based on the results of fatigue driving
status recognition and IoT technology, this
paper designs an early warning system. The
system can realize data collection,
identification, and early warning. When a driver
is detected to be fatigued, the system not only
sends a warning signal to the driver but also
informs other nearby vehicles using this system
through the Internet of Things technology and
manages the operation background.
5. Conclusion
The rapid identification and early warning
of the fatigue driving state are the key to
reducing traffic accidents. Quick and accurate
fatigue identification is a prerequisite for
effective early warning. This study is based on
two-modal data of EEGs and EOGs to identify the
fatigue driving state and extracts multiple
features of the two-modal data for experimental
analysis. Experimental data show that the
fatigue state recognition accuracy of multimodal
data fusion is higher. In the selection of
classification models, deep learning algorithms
have a leading advantage, and the recognition
accuracy is higher than that of machine learning
algorithms. However, considering the real-time
requirements of fatigue state recognition tasks,
this study proposes an FSVM algorithm that can
quickly provide model training. The FSVM
algorithm greatly improves the training speed of
the model without reducing the recognition
accuracy and achieves the expected effect. On
the other hand, based on fast and accurate
recognition results, this article designed a set
of early warning systems based on IoT technology
to extend the early warning information from a
single vehicle to the Internet of Vehicles. When
the driver is in a fatigue state, the system can
not only send a warning signal to the driver but
also notify other nearby vehicles using this
system and manage the operation background
through IoT technology. Regarding the
identification of fatigue status, the next step
of this research will be to improve the accuracy
of identification, and more modal data can be
introduced for comprehensive decision-making. In
an early warning system, when the vehicle speed
is too high and the distance is too large, the
signal between the vehicle and other vehicles is
likely to be weak, and it is impossible to
guarantee the successful warning of other
vehicles. LoRa has the characteristics of long
communication distance, low power consumption,
and low cost, which may be able to solve the
above problems. This is also the content of this
study, which needs further research in the
future.
