The effect of semantic information.

Semantic difference between target and distractor categories facilitates MIT [25]. In [47], authors attribute this facilitation to the categorical distinction of targets and distractors, supported by four processes. First, any type of visual distinction between targets and distractors facilitates tracking. Second, semantic differences in targets and distractors require the attentional system to choose a different strategy for distributing attention to objects and thus facilitate tracking. Third, categorical information might be saved in visual working memory and thus improve the error recovery process. Fourth, there is a mechanism that groups information according to its category, leading to easier tracking of targets among distractors that belong to different categories. The authors conducted six experiments targeting this hypothesis and found that visual distinctiveness is not the only reason that differential category membership of targets and distractors induces facilitation in tracking performance: Intercategorical information is also important. Their results rejected the “change in attentional distribution strategy” hypothesis since targets always attracted more attention than distractors. Eye-tracking data have shown that fixations are directed toward targets; the frequency of fixations landing on the targets increases with enhancing the attentional need for processing the targets [36]. This is why change detection is also easier for targets than distractors in a tracking task [48]. Finally, they argue in favor of the fourth process underlying this effect, i.e., the categorical distinction reduces the interference of the distractor in target tracking, so there is a semantic category-based system that facilitates tracking in similar situations. This suggests that adding common features between targets as shared information may increase MTT algorithms’ performance and improve their accuracy in discriminating targets from other parts of the scene. Effectiveness of a similar idea was proved in the object-recognition domain [49].

To understand which brain area is responsible for semantic category-based grouping, brain activity using fMRI is studied [9]. Subjects in the experiments tracked multiple targets in three scenarios: 1) All targets and distractors were from the same category, 2) targets and distractors were from two distinct categories, and 3) half of the targets and half of the distractors were selected from one category and the others from a different one. Results suggested that the fusiform and pars triangularis of the inferior frontal gyrus are involved in semantic categorization.

Familiarity of targets also facilitates tracking. One can regard this familiarity effect as a kind of semantic distinction. Tracking multiple familiar identities versus tracking unfamiliar ones is compared in [26]. It is postulated that the ease of tracking multiple familiar identities originates from less cognitive load needed to remember the identity of targets. Relating this to the two systems responsible for MIT, location and identification, familiar targets reduce cognitive load since identification of these targets is less cognitively demanding. Familiarity of objects depends on the frequency of their use in trials of a block of experiments. To study familiarity, the identity of targets and distractors are the same in all trials within a block. In the unfamiliarity condition, the identity of targets and distractors always change, and there are no two trials with the same targets and distractors in the block. In [50], authors performed an experiment to investigate whether among familiar objects (animals and man-made ones) there is any bias toward one category or the other. They asked subjects to track multiple objects among animals and artifacts. Although they observed a small increase in identity accuracy for animal targets, they suggested that animal targets do not improve location tracking and that there is no absolute advantage in their tracking over man-made objects. The neural processes involved in tracking familiar targets and unfamiliar ones are examined in [26]. While tracking unfamiliar objects, activity in regions within the attention network, and areas responsible for identification increase. However, activity in areas underlying memory performance increases while tracking familiar objects. Accordingly, it is suggested that MIT is a two-stage process: finding the location of targets in the first stage and identifying them in the second stage. When subjects localize the targets, they extract some key features to identify them and understand what is where. As the subject becomes progressively more familiar to targets, key features can be determined more efficiently while ignoring irrelevant features. This process facilitates finding the loci of targets.

Emotional content of the target items can also affect tracking performance. Li and colleagues performed a multiple–identical-face tracking task to examine the effect of facial emotion on tracking performance [27]. Subjects could track angry faces as targets more successfully than other faces in general. This effect persisted even when another angry face existed among the distractors. This means that processing of emotional expression happens quickly and affects MTT performance. They did not report such effects for a happy expression.

Familiarity and emotion can make both targets and distractors semantically distinct. This type of distinctiveness facilitates perceptual grouping and increases MTT-tracking performance. In fact, discriminating targets from nontargets is easier when utilizing both semantic and appearance information compared to appearance only. Computer-vision algorithms consider appearance features to discriminate targets and distractors, but semantic information has rarely been used for this purpose. Using semantic information improved an algorithm’s performance in an object-detection application [29,30]. Applying a similar approach to MTT algorithms can be beneficial and potentially improve their performance. In addition to semantic categorization, the visual features of objects can also affect MTT accuracy. For example, assigning a single color to all targets and another color to all distractors makes tracking easier [28]. In the following section, we explain how appearance features affect MTT.

The effect of surface features.

Surface features such as color and shape are the most easily perceivable features that help an observer distinguish between objects. A study investigated whether surface-feature information contributes to MOT or if tracking multiple objects is based on spatiotemporal information alone [51]. The experiment consisted of several spheres moving on a floor plane. The floor plane was presented against a black background from a viewpoint angle of 20°. Target spheres were introduced by flashing red at the beginning, after which they started moving. In the meantime, distinct colors were assigned to the spheres for a period of time. An abrupt scene rotation around the axis occurred after motion onset. At the end of the trial, participants had to select the target objects. Their results indicated that brief presentation of object colors around a scene rotation influences tracking performance, and distinct color matching across the scene rotation improved tracking. Swapping the distinct colors after rotation impaired tracking performance. The authors introduced a flexible-weighting tracking account, revealing that spatiotemporal information and surface features are both utilized by the location-tracking mechanism. The two sources of information are weighted according to their availability and reliability. Surface-feature effects on tracking are particularly likely when distinct surface-feature information is available and spatiotemporal information is unreliable [51].

To investigate the extent to which tracking can be improved by surface-feature distinctiveness in object identities, in another experiment subjects are asked to track a subset of objects either unique in color or unique in color and shape [52]. Results showed that tracking is feature based, and there is a limitation in feature binding. Tracking performance improved when objects were distinct in shape or color; however, when the targets were distinct from one another, tracking was not enhanced if distractors shared the target’s colors or digit identities. This finding contrasts with previous studies, where participants were found to have poor memory for surface properties in a MOT task [53]. Participants use a strategy for tracking to set up a competition between the two types of perceptual grouping: grouping on the basis of attentional set (tracking targets versus nontargets) and grouping on the basis of surface features (red objects versus green objects and so on). This was also reported in [54]: When targets and distractors have distinct features, only surface features of targets are maintained in visual working memory; when targets have the same color as distractors, they are more difficult and consume more attentional resources to track.

In [28], authors investigated how much uniqueness of object identities affects tracking accuracy. They asked participants to track four targets among eight objects that were: (1) identical in color (homogeneous condition), (2) of different colors (all unique; red, green, blue, yellow, orange, azure, brown, and pink), or (3) represented by two or four total colors (heterogeneous condition). They found that accuracy was affected by heterogeneity of tracked objects, and color uniqueness of targets and distractors enhanced performance. The similarity between targets is not important; rather, it is the distinction between targets and distractors that is important. This was further shown in [55], by investigating separately the impact of chromatic and categorical distinctiveness of colors on tracking. Circles from two green and blue categories in different hue ranges were used as objects in the proposed MTT task. Results revealed that the varying distinctiveness in color between the targets and distractors significantly influenced tracking performance. Also, tracking accuracy was affected by the chromatic difference between targets and distractors, which may be attributed to the effect of perceptual grouping [55]. Further, in a separate study, fMRI data showed that the putamen and temporoparietal junction (TPJ) may be involved in this color-based grouping [56]. In [57], authors investigated to what extent target-distractor differences in terms of color, contrast polarity, orientation, size, shape, depth, and combination of shape, color, and size influences tracking accuracy. They compared accuracy in these conditions to accuracy in situations in which two targets and distractors share one feature and the others share another feature. Tracking performance was always better in the former case. Results showed that subjects use a range of features to highlight target-distractor differences that facilitate tracking.

Surface features such as color and shape are encoded in early visual-processing areas of the brain [58]. According to the preceding findings, these features undoubtedly influence the processing of targets and distractors during MTT and facilitate tracking. It seems possible that there is a flexible feature-selection method in our brain that selects a minimum number of discriminative features to separate targets and background in general. However, MTT algorithms lack generality and are biased to find the best feature set for only one specific application. We next study how low-level feature and depth influence tracking motion.

The effect of motion features and depth.

Motion information is also important in MTT [59]. To investigate motion information’s effect on visual attention through MOT, a task that added motion to the texture of each object and to the background is designed [60]. The texture inside targets either stayed stationary or moved relative to the direction of the target's motion. This motion could be in the direction of a target’s motion, opposite to the motion of the target, or orthogonal to it. Tracking accuracy was better in the same direction condition than the orthogonal condition and better in the orthogonal condition compared to the opposite condition. These results show that texture motion influences tracking accuracy, but texture speed does not affect it. Thus, when there is a motion conflict between the target’s motion direction and the texture motion, tracking is impaired. This result was also replicated in a 3D scenario. Intrinsic motion of an object is a feature that is not used by MTT algorithms, although this feature could help to discriminate nonrigid and deformable targets from other parts of a scene.

In addition to intrinsic motion, an object’s movement speed and direction are usually set randomly to force the object to have independent motion trajectories in MTT tasks. In some studies, velocity is considered constant to ignore the effect of acceleration [61]. In addition to tracking in a 2D environment, a limited number of experiments were performed on 3D MTT tasks that found an advantage to tracking in 3D compared to 2D; for example, it has been reported that stereopsis effect has a positive impact on MOT [62]. It can even facilitate some related skills similar to playing football. Tracking multiple objects in a 3D environment improves passing decision-making accuracy in soccer players [63]. Using depth information in computer-vision algorithms is beneficial, too [32,33], even though MTT algorithms rarely use binocular vision and depth information.

Simple low-level surface or motion features involve less attentional resources compared to high-level information, which can have a negative impact on MTT performance. In the following section, we focus on how identity processing of objects interferes with location processing.

The interference of identification.

The common resources used for processing location and identity information cause them to interfere with each other [37,64]. In one study, subjects were asked to track four objects from eight identical circles and then select targets and report their identity at the end of each trial. Results showed that tracking identical targets is easier than recalling their identities. It is possible that subjects have an internal name for targets paired with external labels; however, this also assumes that internal names are available for use outside the tracking task. It is also possible that the internal name is available only for the purpose of tracking and is not retained outside that process. This would operate similarly to a local variable in a computer subroutine, which is not available to the program that calls the subroutine [64]. In [65], authors studied to what extent we can recognize identity while tracking multiple objects. They asked subjects to track grayscale human faces with neutral expressions from a Chinese face database. Study results indicated that processing of target identity is a mandatory process that can occur even when it is task irrelevant. Results also showed that performance of tracking different faces was impaired compared to tracking of identical faces, which can be explained by a dynamic identity-location binding for faces and objects by some mandatory processes. The study also demonstrated that tracking and identity processing share the same attentional resources.

When high-level processing such as face identification is involved in MTT, the identification process requires more attentional and memory resources, resulting in lower MTT accuracy and location information processing. In fact, people tend to use optimized features that not only can discriminate targets better but are also encoded and retrieved faster and more easily. This is why in the real world, where a lot of information exists for processing, we are still successful in tracking various objects simultaneously. Some MTT algorithms benefit from an object-detection module to determine all targets in all frames of video. Some use discriminant features to separate targets from background and track them in next frames. However, the brain utilizes a combination of these approaches. We have thus far considered how visual information affects identification and location processing in MTT; the following section explores the influence of other information, such as spatial configuration.

Spatial configuration and the hemifield effect.

To study the role of overall spatial configuration of targets in MTT performance, an MIT experiment is designed in [10], and subjects are asked to track objects with either distinct colors or distinct irregular shapes. They found that the spatial configuration of targets impacts their identification. Preserving the form of a nonrigid polygon (with each target as a vertex) while tracking, the targets can be identified more accurately. This suggests that subjects tend to mentally build a polygon, considering each target as a vertex. They focus on its center of mass overtly and track the vertices covertly. This means that the responsibility for tracking targets is assigned to both hemispheres, which offers a benefit of use in MTT algorithms, such as tracking cars in a highway. Specifically, this strategy could help to detect the location of objects that are partially or completely occluded. Hemifield effect is also studied during MIT [66]. Comparing tracking performance when all targets are moving in one hemifield versus when they are distributed in both hemifields showed that the cognitive resources for tracking are not hemifield specific. Tracking accuracy is higher when targets are distributed across two hemifields, and this bilateral advantage was stronger during MOT than MIT. Tracking is partially independent in two visual hemifields, and the degree of independence is greater in MOT than MIT.

In [67], the authors investigated the effects that speed and proximity of objects to each other have on tracking performance. They used the Planets and Moons Tracking (PMT) paradigm, which consists of a series of dots rotating around local centers and also around a fixation marker, similar to a solar system. Some factors of the experiment (proximity, speed, eccentricity, and number of distractors) can be manipulated independently. The results of this study clearly indicate that speed, proximity, and the number of distractors in the display each influence tracking performance while the other factors are held constant. Thus, models of object tracking need to include mechanisms that are sensitive to these factors. These results suggest that there are at least two mechanisms at play during tracking: One mechanism is sensitive to the number of distractors in the display, while a second is sensitive to target-distractor proximity as well as the speed of the objects in the display [67]. Tracking performance declines with increasing speed and decreasing distance between objects. Overall tracking accuracy is always higher in 3D compared to 2D [68]. According to [69], more attention is allocated when targets are in a crowded situation and have a higher chance of being lost. More precise tracking is needed in such situations, and an increase in accuracy in these cases supports the idea of dynamic allocation of attentional resources.

Occlusion and crowding.

Studies have shown that humans are able to track targets successfully even if targets move behind an occluder or are out of view (due to disappearance) for a short time [70–72]. Although occlusion causes performance to decrease slightly compared to the no-occlusion condition [72], attentional resources are still devoted to occluded or invisible objects [73]. In [74], authors compared brain activities recorded using fMRI while tracking occluded and fully visible objects covertly to unearth neural substrates underlying occlusion handling. They reported that cognitive strategies and mental states behind these two cases differ; they found more activation in four regions of the brain in the occlusion condition compared to no occlusion: inferior parietal lobule, superior temporal sulcus, presupplementary motor area, and precentral sulcus.

Subjects track more successfully when objects simultaneously disappear rather than when they disappear asynchronously for several hundreds of milliseconds and reappear again [70]. Some studies have shown that humans use available and reliable surface features in addition to spatiotemporal information to handle occlusion [51]. To investigate how humans use spatiotemporal features to track objects through occlusion, authors in [75] defined a wall in their MTT task and manipulated location, motion direction, and the side of objects’ reappearance when they go behind the wall and disappear. They reported that MTT performance is better when the target reappearance position is near to its disappearance location regardless of motion direction and whether objects reappear on the same side they disappeared on. In [61] also, authors found that accuracy is higher when objects reappear near their disappearance locus than when they appear on their trajectories. However, a debate persists on whether humans use trajectory information to handle occlusion. Evidence supports both ideas [46]. Authors in [76] argued that extrapolation and motion information are used more frequently when tracking two targets and less frequently while tracking four targets. In the computer-vision domain, most MTT methods use a motion-estimation module, which significantly increased the accuracy of algorithms.

Crowding is another important factor in an MTT task. When the number of objects increases in the scene, the distance between objects decreases. This factor is the primary human limitation in MTT [20]. Performance falls by decreasing spatial separation between targets [77], and in such situations, when the possibility of confusing targets increases, it is suggested that human subjects benefit from rescue saccades (the saccades toward the targets that are in a critical situation) to avoid wrong target association [78]. Studies have shown that subjects utilize working memory and allocate more attentional resources to crowded parts of a scene [69,79]. In addition, they tend to locate their gaze close to the targets that are in a crowd. This way, the crowd is projected to the fovea and can be handled easier because of the fovea’s higher spatial resolution [80]. In general, this spatial limit in MTT is distinct from the attention-capacity limit [81].

Attention-based MTT methods.

Humans are unable to process all incoming visual information simultaneously. To overcome this limitation, they use attention as a mechanism to select what to process. The overtly attended region is projected onto the fovea (a part of the retina with the highest spatial acuity and the highest density of photoreceptors), and the surrounding areas are projected onto parts of the retina that are far from the fovea, which have a lower density of photoreceptors. Many researchers have studied attention, and different models have been proposed to predict which part of an unseen image will be attended to by humans. Many attention models have been developed in the past, and reviewing them all goes beyond the scope of this work. Refer to [94–96] for comprehensive reviews on attention modeling.

A fundamental concept in modeling attention is a center-surround mechanism, suggesting that an item that is highly different from its surround is salient and attracts attention. This mechanism has been used in several algorithms. For example, authors in [23,97] utilized this mechanism to propose a discriminant tracker for a single object. They chose and used features with the highest ability to discriminate a target from its surrounding background from a feature pool. For this purpose, they applied features on both center and surround. Then, the Kullback–Leibler divergence between probability distributions of center and surround is calculated to measure the ability of features that discriminate these two. In [38], another strategy is proposed to benefit from both center and surround information to facilitate tracking. The author uses a swarm of tracking windows, each capable of tracking a part of a single moving target. Windows are assigned to corners of a target after applying a corner-detection algorithm in the initialization phase. Therefore, some windows contain some parts of the target, and others contain its surround. Windows are tracked separately, and the patch with the highest normalized cross-correlation (NCC) is considered to be the new position of the target. Motion direction, speed, and the size of the target were also inferred from corners. Target motion is calculated according to mean motion of windows. To handle changes in target size, the associated motion of windows in right versus left and top versus bottom is examined. Changes in swarm motion can be interpreted as changes in size. Outlier windows that have inconsistent motion regarding swarm motion are dropped, which reduces window population after several steps. In this case, swarm is reassigned to the targets. This study uses corner features of objects to follow their changes in size. However, in cognitive studies of MTT while subjects are instructed to track targets in a 3D environment, it is an open question how they treat changes in targets’ size due to perspective.

Attentional mechanisms are usually characterized as either bottom-up or top-down processes. Bottom-up attention is stimulus driven and unconscious, whereas top-down attention is goal driven and conscious. Some algorithms use these mechanisms to develop search capabilities. They usually use bottom-up attention to extract features and top-down processes in order to find the target’s new position (the most similar object according to extracted features) in the new frame. Features extracted in the bottom-up phase can be discriminant features [23,97]. In the top-down phase, some algorithms search the neighborhood of the current position of the target in the next frame to find the most similar part. These two steps are repeated one after another, which means that target features are updated in their new positions to easily track appearance changes in the target. In other words, in such discriminant trackers, there is less bias towards using target representative features. Instead, features with a high ability to discriminate between the target and background are used, which is useful for handling both the target`s variable appearance and variable background challenges. For tracking multiple targets, this strategy is extendable.

Another method considers spatiotemporal attention mechanisms using a framework based on dynamic CNNs for MTT [98]. This method solves two MTT challenges. Using a dynamic CNN helps handle computational complexity, and using spatiotemporal attention helps avoid missing objects in case of the occlusion of multiple targets. The CNN-based framework has some shared layers and devotes a separate branch to each tracker. The shared layers encode the features of the whole frame and do not change during the tracking process. The branches have the same structure, while each one is trained on a separate object as an SOT; a new branch is devoted to a newly defined target, and, when a target is removed, its corresponding branch is removed as well. Each branch consists of a visibility map that determines nonoccluded regions, an attention map to weigh the feature map, and a binary classifier to separate the object from its background. This appearance model is trained online using new and historical samples via the backpropagation algorithm. Temporal attention is also considered to weigh the previous correctly detected samples against the new ones according to the amount of occlusion. The more a sample is occluded, the lower its weight.

Authors in [99] focused on spatiotemporal continuity by checking for inconsistencies in spatial coherence according to psychological findings suggesting that spatial information plays a dominant role in preserving target identity. Their method considers potential associations of targets in the spatial domain and evaluates appearance, motion, and other features in the visual domain as inspired by the famous two-stream hypothesis about what (object) and where (location) attentional pathways [100].

Among different approaches reviewed in [101], attention provides a suitable method for recognizing targets by discriminating them from the background and from each other. This helps handle important challenges, such as variable target appearance and background variability. But, due to its adaptability to variation, it suffers from occlusion. When a target is gradually occluded by an occluder, the algorithm may tend to accept it as a change in target appearance and thus miss the target. A successful solution to avoid this involves using memory to retain appearance information of the targets and limiting the range of variations in target-appearance changes.

Memory-based MTT methods.

Memory is the second cognitive mechanism with significant impact on MTT performance. To find the position and identity of targets at each moment, one needs to remember them. This helps to overcome occlusion challenges. In the case of partial occlusion, it helps to detect the parts that belong to the target and to exclude the occluder. In the case of complete occlusion, after reappearance of a target from a complete occlusion, we can remember which object was the target. Memory also helps retain different appearances of a target and overcome the variable target-appearance challenge. According to the Atkinson–Shiffrin memory model (ASMM) [102], there are three memory stages: sensory memory, short-term memory, and long-term memory. A study used this model to handle sudden changes, such as illumination changes and occlusion, in the proposed MTT algorithm [103]. The model is based on compressive tracking that uses sparse and reduced discriminant features to find the target in a new frame. The authors also applied a weighting method to emphasize the nearer candidates for targets (according to the euclidean distance between the target in the previous frame and the candidate in the new frame) and to lower the contribution of others. This approach leads to higher accuracy when the target is similar to the background.

In [104], authors used the aforementioned memory model to propose a single target MUlti-Store Tracker that can memorize target appearance. This method supports both translation and scale invariance. A forward-backward tracker [105] is applied to two consecutive frames to identify any failure in tracking, i.e., the tracker estimates the target position in the first image according to its determined position in the second image. The amount of displacement is expected to be small. Comparing the estimated position with the real position identifies the best key points to successfully track the target; key points are used to find the best match for the target, to determine its position in the new frame according to the euclidean distance, and to reject outliers. Occlusion handling is also possible. A small number of key points matching the background in the targets’ bounding boxes means that no occlusion is happening. As the number of key points matching the background approaches the number of target key points, the target is more occluded. The target is remembered and stored in short-term memory when it is not occluded.

Forgetting is also used for descriptors inspired by how humans lose information when there is no attempt to retain it. As a feature is recalled more, forgetting becomes harder. Another memory-based SOT is proposed with adaptability to variation in target appearance and environment [106]. This proposed memory model has three components: ultra-short-term memory, short-term memory, and long-term memory, each associated with the processes of encoding, forgetting, and remembering. Both of the above-mentioned algorithms can be extended to track multiple targets. Targets and background usually change over time in a natural environment. A challenge for computer-vision algorithms is successfully keeping track of targets in these situations. On the other hand, stimuli used in cognitive studies of MTT are too simple and lack any changes that may happen naturally. A productive area for future study is how humans deal with this challenge in MTT.

The memory module in an MTT algorithm is useful for handling challenges in target tracking, such as occlusion, variability in target appearance, and data association. In tracking more than one object, the role of memory becomes even more important. However, a memory-based algorithm needs to establish some critical parameters, such as what to remember and when to forget something. The importance and necessity of memory in algorithms motivated some studies (in particular, those using deep neural nets as their main tool) to propose LSTM (Long–Short-Term Memory) networks.

MTT methods based on ANNs.

ANNs are commonly used in computer vision. They are inspired by the human nervous system; each neuron in an ANN receives inputs from one or more neurons, applies a simple function on the weighted sum of inputs, and emits its output to others. Different attempts have been made to improve neural nets according to biological neural-processing evidence. These efforts have led to convolutional neural nets, recurrent neural nets, and spiking neural nets as well as networks with various architectures (e.g., the Hopfield network).

In the context of tracking, neural nets have been widely adopted. In [107], a five-layer network inspired by the primary visual cortex is proposed. Three layers perform object recognition and present a biologically inspired appearance model. The remaining two layers strengthen the discriminative ability of the model in distinguishing the target from the background. This architecture also includes layers for whitening, coding (which uses discriminative dictionary learning methods), rectification, normalization, and sum pooling (making the model find global features). This network was proposed to model appearance when a particle-filter algorithm is used for tracking. The weights for particles are calculated according to the similarity of the learned target and candidates. The particle with the highest weight is considered to be the tracking result in the current frame. To handle target variation during the video, a set of target templates are used, and a weight is assigned to each one to define the current appearance of the target. Authors in [108] postulated that having a large amount of auxiliary data is enough to learn a CNN (in an online manner) that can track a target robustly without offline training. They proposed a lightweight, two-layer feedforward convolutional network tracker (CNT) accordingly and ignored pooling layers in the architecture of their net in order to keep high-resolution spatial features and their precise positions. A particle filter then estimates the exact position of the target in the new frame.

The methods mentioned above are all SOTs. Neural-net–based algorithms for MTT, however, are rare. For example, [109] benefits from the motion-history image, which is the difference between the current frame and historical images. However, their approach cannot detect stationary objects. Therefore, another feature is also used to strengthen object detection and recognition. A number of filters are learned using a convolutional restricted Boltzmann machine in an unsupervised manner. Using features and motion-history information, a feature map is produced for each frame. Using this method, all objects in the scene are tracked, and this information is used to detect and recognize objects in a scene.

Another study proposed an end-to-end recurrent neural network (RNN) approach with an LSTM for MTT that can handle various challenges, especially the unexpected entrance or exit of objects, which happens in a real scenario [110]. This is a challenge for algorithms that is ignored by almost all cognitive studies of MTT. The RNN is responsible for temporal prediction, and the LSTM is responsible for handling data association. The proposed method is entirely data driven and does not need any prior knowledge. Required data is generated by sampling from a generative model. A multidimensional state space is used concurrently to determine the states of all targets with different types of variables and desired numbers of outputs, each corresponding to a single target. In robotics, authors in [91] used an RNN (particularly, a deep LSTM network) with the help of heading estimation and spatial information to identify robots with the same appearance in robot collaboration tasks.

Recently, deep reinforcement learning (DRL) has also been shown to be an innovative approach for demonstrating object localization, active object tracking, and MOT [111,112]. For example, [113] used a DRL method to locate targets in the new frame considering the collaborative interaction of targets and environment based on detection results of targets in the previous frame. For this purpose, they combined a prediction network (responsible for predicting the location of targets) and a decision network (responsible for feature extraction and finding optimal results regarding interaction of targets and the environment). In this way, they initialize, delete, or update information related to all targets in each frame.

In general, ANNs have the potential to apply to cognitive mechanisms and are shown to be effective for tracking more than one object. However, problems remain (e.g., the requirement for a large amount of data, instability, etc.), which sometimes makes ANNs unsuitable for some real-world scenarios.