1: Image extracted from [6]
The above image shows the general stages of a pattern recognition system. Feature extraction makes it easy for patterns to be matched to a particular model. This stage precedes classification. As a general rule, pattern recognition systems must easily and quickly recognize familiar patterns. For instance, for a computer vision system to be deemed successful, recognition of objects must be accurate even when the object is viewed from a different angle or when the object is partially obstructed [6]. Different variables are used to represent the features in a pattern. These variables may take a discrete, continuous or discrete binary form. When combined, the identified features form a feature used in detecting patterns from new input data.
In classification, the final stage of pattern recognition, models that generate identified patterns are recovered and the input data classified into the respective models [6]. The most crucial element in a patter recognition system is the classifier which sorts patterns based on their respective models. Classification requires supervised learning since appropriate labels must be assigned to datasets used in training. Clustering of input data, on the other hand, is unsupervised and can be done using machines. Using known patterns from training samples, a pattern recognition system sorts input data into the established categories/pattern groups [6]. Once a pattern is matched to its model, the pattern recognition system makes the required decision. This decision-making phase is referred to as the post-processing phase. Pattern recognition spans across various niches in machine learning such as speech recognition, retrieval of information, face detection, automatic medical diagnosis etc., where instances with repetitive, observable features occur [24]. The pattern recognition process is shown in the image below.
2: Image extracted from [6]
3.1.2 Design process of a pattern recognition system
The design cycle of a pattern recognition system begins with data collection [6]. Researchers first determine the appropriate data of the optimal size for training and testing of the system [6]. Based on the collected data, features are selected for each pattern that the system is intended to detect. Selection of features requires prior knowledge of the characteristics of each pattern [6]. Based on the selected features, appropriate models are determined and created [6]. The model choice is heavily dependent on the data collected. For most pattern recognition systems such as those used in speech recognition, the appropriate model is based on probability of the input data matching the available sample [6]. Training of the pattern recognition system begins after construction of models. Using the data collected, the feature extraction and classification components of the system are trained to identify and cluster features before classifying them into the respective models [6]. The main objective of training is to develop a classification system for the pattern recognition system. Accuracy of the classification component depends on the purpose of the pattern recognition system. A trained system is evaluated for accuracy by comparing the output data with samples of the expected output.
3.2. Applications of HITL Approaches
3.2.1 Computer vision
Computer vision is applied in development of face recognition systems used in law enforcement agencies for identification of suspects [1] and in development of facial unlock systems in smartphones. Use of the human-in-the-loop approach in the data labeling and annotation and the evaluation phase proves useful in the design process [3]. High performance computer vision systems with fine-grained recognition can only be developed with the input of humans [20] [26]. Crowdsourcing the annotation process allows researchers to dedicate more time to the data collection and training phases of design [17]. Human-in-the-loop approach is also used in the development of person re-identification systems to differentiate people with similar appearances in video surveillance [21]. By using human feedback, the machine’s classification component can continually be improved [22] [23].
3.2.2 Gaming
Data collected from games is very valuable in the design of pattern recognition systems. In the popular Peekaboom game, players are required to identify objects in the game. Machine learning researchers have leveraged the crowd-in-the-loop approach to use data from human players in training image recognition systems [7]. Another game whose data is used by machine learning experts is Bubble. In their paper, Deng et al. explore how Bubble aids in pattern recognition and machine learning [8].
3.2.3 Medical services
As highlighted in Section 2, human-in-the-loop approach in the development of pattern recognition systems applied in various medical fields [5]. Holzinger explores the various applications of human-in-the-loop in clustering of data, protein folding and k-anonymization [5]. Doctors and research experts in various fields can participate in development of pattern recognition systems designed for predicting protein folding structures [5]. The use of human-in-the-loop approach is also employed in developing whole body MRI pattern detection systems used for diagnosis of different illnesses [25].
4. HITL Approaches for Pattern Recognition Systems Design
4.1 Feature Extraction and Selection
Feature extraction and selection, as initially mentioned, is a key part of the design of a pattern recognition system. Embracing the human-in-the-loop approach in feature selection is useful due to the developed human pattern recognition abilities [18] [19]. Use of the crowd-in-the-loop approach during the feature extraction phase enhances the design process since leveraging large numbers of people helps in the identification of extra features that might be missed by a single individual [8]. Deng et al. explore the advantages of embracing the crowd-in-the-loop approach in developing recognition systems [9]. In the paper, the game Bubble is explored, where human players are required to identify the category of a heavily blurred image [9]. A player can choose to expose part of the image, or a bubble, at a penalty. The bubbles selected are recorded and used to improve the machine’s recognition system [9]. When a game player picks a bubble, a machine learning researcher can categorize the bubble as a feature of the particular image and use the knowledge to train the pattern recognition system.
4.2 Labeling or Annotating Samples
The human-in-the-loop approach also plays a crucial role in the labeling and annotation of samples. Scheirer et al. conduct various tests to identify the benefits from crowdsourcing the annotation process. In a period of seven and a half weeks, 337,932 annotations are collected from 3,250 researchers via the TestMyBrain website [4]. Based on the machine’s results, the crowdsourced annotations, which are far more descriptive than the typical labels used in supervised learning, exceed the set expectations of the experiment conducted [4]. The face recognition system makes significant progress in identifying faces in images. Crowdsourcing the annotation process improves the perception of the machine [4]. Based on the paper, sufficient evidence on the impact of human-in-the-loop approach in annotation and labeling of datasets can be drawn. As seen in Doris et al. in the paper on accelerating machine learning through Helix, use of the crowd-in-the-loop approach also reduces the total time spent on labeling [16].
4.3 Performance Evaluation
The performance of training efforts on machine learning systems also benefits from the human-in-the-loop approach. Holzinger’s paper on applications of HITL in the medical field discusses the benefit of engaging medical professionals in the evaluation of decisions made by pattern recognition systems applicable in the field [5]. Given the high stakes involved in the medical field, professionals always have to review the decisions output by pattern recognition and machine learning systems before execution of the output. Crowdsourcing the performance evaluation task enables machine learning researchers to appropriately modify the system to meet the experts’ standards. Branson et al. also credit the human-in-the-loop approach for success in computer vision systems where humans evaluate the accuracy of the system in recognition of image [20].
5. HITL Approaches for Deployed Pattern Recognition Systems
5.1 Feedback on Predicted Labels
Holzinger also explores the role of HITL in deployed systems such as the protein folding prediction system described in the paper [5]. Medical professionals from various fields are involved in the crowd-in-the-loop approach to provide input on the predictions made by the machine. With input on predicted labels from experts, the machine can gather interactive feedback from the user. Active learning occurs when the machine learning system can interactively gain input from human agents or other sources to obtain the expected results. In crowdsourced feedback, data is drawn from a pool and the machine’s ‘understanding’ is measured based on the performance. The machine attempts to put labels on the datasets and interactive users measure its accuracy. Feedback is then provided to the machine by employing a human-in-the-loop approach. The machine’s system uses the feedback input to improve its labeling attempts. Active learning is observed in Deng et al. paper, where machine learning experts improve the pattern recognition system by comparing the machine’s output to the human game players data.
In Raghavan et al. paper, various techniques applicable in active learning are examined. They include active learning augmented with feedback, use of feature feedback before active learning and the use of feedback after active learning [10]. In the paper, use of human teachers, also referred to as users, is slower than use of a source, referred to as oracle, for information. Individual human teachers have imperfect information whereas oracles have more credible information [10]. However, crowdsourcing significantly improves the overall credibility of feedback given to the machine. In Endert et al. paper on model steering, flexible and expressive interactions between the machine and humans during clustering increases the success of visual analytic tools [13]. Endert et al. explore how human clustering improves the visual analytic system’s labeling [13]. By providing continuous feedback, the visual analytics system increasingly gets better with time [13]. In a different paper on visual text analytics, Endert et al. credit involving humans in the process as critical to successful operation of the systems [14]. In this paper, a visual text analytics system is developed using the HITL approach [14]. Human agents provide feedback to the machine on how to update existing models based on new labels the human agents identify [14]. Similar to the visual analytics system, human involvement in the development process improves pattern recognition in text [14].
5.2 Feedback on Sample Relevance/Ranking
In most images, document retrieval and person re-identification systems, training of the machine on relevance is crucial to the improvement and increase in the system’s efficiency. Use of a human-in-the-loop approach to provide feedback to the machine can prove to be beneficial. In Wei et al. paper, the need for a human-in-the-loop approach is observed in development of mammogram retrieval systems used in diagnosis of cancer [11]. Medical experts continually provide relevant feedback to the machine to aid in improved learning [11]. Yuatin attributes success of personal re-identification systems used to monitor pedestrian movement to supervised learning where ranking of re-identification results is guided by humans [12]. According to Cao and Ai, use of the human-in-the-loop in a computer vision system improves the machine by providing feedback on the relevant samples related to the task the machine is required to meet [15]. In the computer vision system, humans assist the machine in ranking image identification results based on similarities [15].
5.3 Interactive Clustering/Segmentation
Over time, developed pattern recognition systems need to be readjusted to match changing conditions which influence pattern features. As a result, human-in-the-loop plays the important role of interactively aiding the machine in forming new clusters based on changing input. In the case of protein folding prediction systems [5], changing protein patterns based on DNA alterations of the biological agent under study demonstrates the need for continued human-in-the-loop approach in pattern recognition systems applied.
6. Discussion and Open Issues
Based on the literary works analyzed in Section 2 of this paper, the role of the human-in-the-loop approach in pattern recognition systems is significant. However, there remains several issues underlying the approach. One is the bias and expectation that human input will be perfect, which unfortunately, is not the case in reality. Measures of ensuring bias and unwanted discriminative traits are not introduced into the machine learning systems are yet to be fully explored. In addition, the machine learning experts hope to improve learning techniques to the point where machines develop similar capabilities to humans. However, for machines’ abilities to transcend human capabilities, new approaches besides the human-in-the-loop approach will have to be devised. This creates a challenge that machine learning experts can expect to come across at a point in the future. Currently, there exists a gap on how to ensure human biases are not introduced into machine learning techniques, providing a solid topic for research.
Works Cited
[1]. S. Ali et al. Interactive Retrieval of Targets for Wide Area Surveillance, ACM Multimedia, pp. 895-
898, Oct. 2010.
[2]. J. Wang, Y. Wang, and Q. Lv. “Crowd-Assisted Machine Learning: Current Issues and Future. Directions.” Computer, vol. 52, pp. 46–53, Mar. 2019. doi:10.1109/mc.2018.2890174.
[3]. A. Kovashka, O. Russakovsky, L. Fei-Fei, and K. Grauman. “Crowdsourcing in Computer
Vision”. Foundations and Trends in Computer Graphics and Vision, vol. 10, pp. 177–243, 2016
[4]. W. J. Scheirer, Anthony S, Nakayama K and Cox D. “Perceptual Annotation: Measuring Human Vision to Improve Computer Vision”, IEEE, Berlin, 2013.
[5]. A. Holzinger. “Interactive machine learning for health informatics: when do we need the
human-in-the-loop?” Brain Informatics, vol. 3, pp. 119-131, 2016.
[6]. D, Richard O., Peter E. Hart, and David G. Stork. Pattern classification. John Wiley & Sons, 2012.
[7]. L. Von Ahn, R. Liu, and M. Blum. “Peekaboom: a game for locating objects in images.” Proceedings of the SIGCHI conference on Human Factors in computing systems. ACM, 2006
[8]. C, Alvaro HC, and Freddy Lecue. “Human-in-the-loop feature selection.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 33, 2019.
[9]. J. Deng, J. Krause, M. Stark, and L. Fei-Fei. “Leveraging the wisdom of the crowd for finegrained recognition”. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), vol. 38, April 2016
[10]. R, Hema, O. Madani, and R. Jones. “Active learning with feedback on features and instances.” Journal of Machine Learning Research . vol .7, pp. 1655-1686, Aug 2016.
[11]. W. Chia-Hung, Y. Li, and P. Jung Huang. “Mammogram retrieval through machine learning within BI-RADS standards.” Journal of Biomedical Informatics. Vol. 44, pp. 607-614, 2011.
[12]. Lin, Yutian. Deep learning approaches to person re-identification. Diss. 2019.
[13] A. Endert, P. Fiaux, and C. North. “Semantic Interaction for Sensemaking: Inferring Analytical Reasoning for Model Steering”. VAST ’12, 2012.
[14]. A. Endert, P. Fiaux, and C. North, “Semantic Interaction for Visual Text Analytics”. CHI ’12, pages 473-482, 2012. https://doi.org/10.1145/2207676.2207741
[15]. C. Cao and H. Ai, “Facial similarity learning with humans in the loop,” Journal of Computer Science and Technology, vol. 30, pp. 499-510, 2015
[16]. X, Doris, et al. “Helix: accelerating human-in-the-loop machine learning.” Proceedings of the VLDB Endowment. Vol. 11, pp. 1958-1961, 2018
[17]. S. Branson et al. “Visual recognition with humans in the loop.” European Conference on Computer Vision. Springer, Berlin, Heidelberg, 2010.
[18]. F. Fleuret. “Fast Binary Feature Selection with Conditional Mutual Information”, Journal of Machine Learning, vol. 5, pp. 1531-1555, 2004.
[19]. I. Guyon. “An Introduction to Variable and Feature Selection”, Journal of Machine Learning Research. vol. 3, pp. 1157-1182, 2003.
[20]. S. Branson et al. “The ignorant led by the blind: A hybrid human–machine vision system for fine-grained categorization.” International Journal of Computer Vision. vol. 108.pp. 3-29, 2014.
[21]. M. J. Metternich and M. Worring. “Track based relevance feedback for tracing persons in surveillance videos”, Computer Vision and Image Understanding. vol. 117, pp. 229-237, 2013.
[22]. A. Das et al., “Continuous adaptation of multi-camera person identification models through sparse non-redundant representative selection”. Computer Vision and Image Understanding. Vol. 156, pp. 66-78, 2017.
[23]. V. Gouaillier and A Fleurant. “Intelligence Video Surveillance: Promises and Challenges” CRIM Research, London, 2012
[24]. S, Priyanka, and M. Kaur. “Classification in pattern recognition: A review.” International Journal of Advanced Research in Computer Science and Software Engineering. Vol. 3, 2013.
[25]. Q., Susana, et al. “Whole body muscle MRI protocol: pattern recognition in early onset NM disorders.” Neuromuscular Disorders. Vol. 22, pp. 68-84, 2012.
[26]. D. Jia, J. Krause, and L. Fei-Fei. “Fine-grained crowdsourcing for fine-grained recognition.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2013.
[27]. S. Amershi et al.: Power to the People: The Role of Humans in Interactive Machine Learning. AI Magazine 35(4), 105-120 2014.
[28]. X. Zhou and T. S. Huang ―Relevance Feedback in Image Retrieval: A comprehensive review‖, University of Illinois, 2010.
For this short paper activity, you will learn about the three delays model, which explains…
Topic : Hospital adult medical surgical collaboration area a. Current Menu Analysis (5 points/5%) Analyze…
As a sales manager, you will use statistical methods to support actionable business decisions for Pastas R Us,…
Read the business intelligence articles: Getting to Know the World of Business Intelligence Business intelligence…
The behaviors of a population can put it at risk for specific health conditions. Studies…