Available online at www.sciencedirect.com
Procedia Computer Science 00 (2022) 000–000
www.elsevier.com/locate/procedia
2022 Annual International Conference on Brain-Inspired Cognitive Architectures for Artificial
Intelligence: The 13th Annual Meeting of the BICA Society
PeC-HiCA: A Perception Centric Human-in-loop Cognitive
Architecture
Amit Kumar Mishraa,∗
a University
of Cape Town, Cape Town, South Africa
Abstract
A layered-model of mammalian cortex is an accepted one. One of the major challenges of using this architecture in real-life usecases is the lack of reliable implementations of the symbolic and cognitive layers. On another parallel stream, the adoption of
cognitive and artificial intelligent (AI) solutions in industries has been slow. One of the reasons for this is the fact that the current
generation of AI architectures do not sufficiently take care of the perception-levels of the human operator. In this paper, we present
an architecture that solves both of the above challenges. We propose a cognitive architecture where a natural-human in loop replaces
most of the symbolic and cognitive layers. At the same time we propose the use of instrumentation to measure the perceptual level
of the human in real time. This enables us to bridge the symbolic non-symbolic gap to some extent as well. Lastly, the perceptioncentric block also adds a novel attribute to the existing AI architectures. Current generation of AI algorithms do not incorporate
the perception of the users as an integral part of the training and operation of the algorithm. To our knowledge, this is the first
time user-perception is an integral part of the algorithm. We call this architecture Perception-Centric Human-in-loop Cognitive
Architecture (PeC-HiCA). PeC-HiCA is not only a more implementable cognitive architecture, it will also be more successful in
industrial applications.
© 2022 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the 2022 Annual International Conference on Brain-Inspired Cognitive Architectures for Artificial Intelligence: The 13th Annual Meeting of the BICA Society.
Keywords: BICA; Perception; Human-in-loop; Perception-Action.
1. Introduction
A six-layered model of neuro-cortex [1], as popularized by Fuster [2], has been popular among the brain-inspired
cognitive architecture (BICA) community. Because, this is an elegant and, yet, simple model to explain the emergence
of cognition in brain. Modern neurological research also validate the model to a large extent [3, 4, 5]. This also models
the symbolic-non-symbolic [6] parts of brain in one model. Higher layers of neuro-cortex are hypothesized to model
∗
Corresponding author. Tel.: +27-21-650-2794.
E-mail address: akmishra@ieee.org
1877-0509 © 2022 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/)
Peer-review under responsibility of the scientific committee of the 2022 Annual International Conference on Brain-Inspired Cognitive Architectures
for Artificial Intelligence: The 13th Annual Meeting of the BICA Society.
2
Amit Kumar Mishra / Procedia Computer Science 00 (2022) 000–000
the emergence of cognition and consciousness. Hence, having a realistic and holistic model representative of all the
six layers is deemed to bring us closer to what Penrose calls strong artificial intelligence (AI) [7].
Implementation of the complete six-layered model is challenging. We shall discuss three of the major challenges
in front of us. First of all there is the challenge of implementing the upper layers of prefrontal cortex which attribute
to the cognitive functions of the brain. This is partly because of the lack of the exact understanding of the operation of
these higher layers. The second level of challenge comes in fusing the non or sub-symbolic layers with the symbolic
layers. This is an accepted challenge where a range of work has been happening starting from mask-based approach
[8] to transformer network based approaches [9]. The third level of challenge is around integrating the action column
of the layers in a perception-action way of looking at the layers. This is partly due to the fact that the reactions
of humans to certain situations are highly subjective. Without integrating the subjective-reaction it is impossible to
implement a truly cognitive architecture. The challenge of modelling human-perception and integrating it into an AI
implementation is also crucial from more immediate incorporation of AI in industries. Due to various challenges,
both technological and ethical, a completely autonomous AI system will take some time to be the state of affairs in
industries. Human-in-loop AI systems would be the realistic solution where there will be human operators [10, 11].
Hence, modelling of human-perception and integrating it into the AI architecture is of high interest to applied-AI
researchers [12, 13, 10]. However, these are preliminary approaches without having a modus operandi on how to
measure perception of the operator or user and incorporate it into the AI architecture.
In this paper, we shall present a cognitive architecture where we address two of the challenges mentioned above.
We argue that with a well-designed human-in-loop AI architecture we can replace the upper layers of the prefrontal
cortex with a real human. To make this architecture work more realistically we argue that modelling human-perception
properly is crucial. We, then, present a real-time architecture which can measure user-perception in real-time and
incorporate that into the operation of the cognitive architecture.
Rest of the paper is organized as follows. In Section 2, I shall articulate the challenges of current perception-action
cognitive architectures. This section will also describe how replacing the upper layers of a cognitive architecture with
a natural-human may reframe many existing AI-solutions in the perception-action cognitive architecture. Section 3
describes the use of bringing perception into the architecture. In this section we shall discuss the perception-centric
architecture in detail. It also presents implementation notes. Section 4 concludes the paper. In this section I shall also
list some of the challenges this architecture may possess and few of the future pieces of work.
2. Human-in-loop Architecture
Layered processing of information in mammalian brain is a well accepted model. Figure 1 shows the model popularized by Fuster [2]. In engineering fraternity, cognitive architectures developed by Haykin [14, 15] have been
inspired by Fuster’s model. It is an elegant model which is both simple to understand and is very close to the way
human brain works.
In spite of being an elegant model, this model has multiple challenges when it comes to implementing it to create
artificial cognition in artificial systems. The infamous challenge of linking symbolic with non-symbolic layers [6] is
one of the foremost ones. Similarly, the feedback paths from the perception vertical to the action vertical and vice
versa are equally challenging to implement in a realistic engineering system. It can be noted that these feedbackpaths are crucial in the emergence of cognition as they enable the creation of internal information (independent of the
information extraction from the sensory inputs).
However, the non or sub-symbolic algorithms have become really powerful because of the accelerating advances
in the field of deep learning. Figure 2 delineates the architecture into two parts, viz.
• the sub-symbolic parts which are relatively easy to implement using deep learning networks, and
• the symbolic and cognitive layers which are extremely challenging to implement as a module.
It should be acknowledged that a lot of development is happening in terms of endeavours to implement the symbolic
and cognitive blocks. However, in the meantime, a pragmatic architecture can use a natural-human expert or user to
perform the tasks expected to be performed by the symbolic and cognitive blocks. Having a human-in-loop (HIL) not
Amit Kumar Mishra / Procedia Computer Science 00 (2022) 000–000
3
Fig. 1. Schematic model of cortical perception-action layers as presented by Fuster[2]. The inter layer feedback paths are specifically noteworthy.
These feedback paths are difficult to incorporate using computers. These might be one of the origins of the emergence of cognition and consciousness.
Fig. 2. Schematic model of cortical perception-action layers as presented by Fuster[2]. In this the non-symbolic layers have been marked. These
are the blocks that are implementable using existing machine learning and deep learning architectures. The symbolic and cognitive layers are more
difficult to implement. Even when they are implemented combining these two parts has been an existing challenge for AI researchers.
only makes the implementation of all the layers of the cognitive model of brain possible, it also makes it easier for
industries to adopt a cognitive architecture. Because, this will enable easier adherence to the AI Regulations [16].
Lastly, this will also make the bridging of symbolic sub-symbolic layers easier. Figure 3 shows the architecture
with the role of the HIL marked with blue borders. The feedback from the human in loop to the sub-symbolic layers
is shown using the blue arrows. To incorporate this, we can measure the human perception levels and quantify them.
This, in turn, will be used as feedback to scale the networks in the sub-symbolic layers.
Amit Kumar Mishra / Procedia Computer Science 00 (2022) 000–000
4
Fig. 3. Schematic model of cortical perception-action layers as presented by Fuster[2] with some of the layers replaced by a natural-human. To
retain the cognitive nature of this architecture, there needs to be a feedback from the human to non-symbolic layers (represented by the blue arrows).
This requires quantitative measurement of human perception.
To summarize, a HIL based cognitive architecture will
• move the burden of implementing symbolic and cognitive layers to the human user/expert;
• enable an elegant bridging of symbolic sub-symbolic layers through measurement of human perception levels;
and
• enable easier adherence to the AI Regulations.
3. Perception-Centric AI
Human perception is an important part of cognition [17]. Perception decides how a human operator or user shall
react to certain output shown by a machine learning system. However, human perception levels (and, hence, the
productivity) are highly dependent on a range of factors starting from emotional state [18] to indoor environment
[19]. Perception level is even more important in industrial environments where a small level of change in perceptual
interpretation by the operator can cause huge loss. In some industries like oil and gas this is well modelled through
frameworks [20]. In other industries it may not be so formalized but the effect of human perception-based errors are
well acknowledged.
Figure 4 shows the architecture of the proposed perception centric HIL cognitive architecture (PeC-HiCA). Input
about the state of the environment, as gathered by sensors, are fed to the first layer which is sub-symbolic. Machine learning algorithms, especially deep learning networks are excellent pattern identifiers (either known patterns or
anomalies). The patterns are then sent to the next layer which analyzes these patterns based on ontological algorithms.
This follows a DIKW-type flow [21]. The outputs from this are presented to the human in loop who is replacing
the symbolic and cognitive layers. The human expert/user reacts and sends commands for actuation or other changerelated commands. However, unlike usual systems, these commands are interpreted through another ML-based block
which gets feedback about the perceptual level of the user. Information from the perceptual information measurement
block are also fed back to the ontological and ML blocks in the perception arm of the architecture. The perceptual
information can be incorporated into the existing ML-algorithm in various manners. A trivial way of incorporation
can be the scaling of weights whereas a more bio-inspired methodology would be through the use of masks [8].
Amit Kumar Mishra / Procedia Computer Science 00 (2022) 000–000
5
Fig. 4. A human-in-loop AI architecture where the human partially replaces most of the symbolic and cognitive layers in Fuster’s BICA. The
symbolic non-symbolic gap is bridged using a real-time measurement of human perception levels and using that in the non-symbolic layers of the
architecture.
As the last part of this section we shall discuss some methods of measuring the perceptual level of the human in
loop. The measurement of perception-level of the human user/operator is an emerging area of research [22]. However,
many of the measurement methods are intrusive. The following would be some of the the non intrusive ways to
measure human perception and attention which can easily be implemented in an industry or work environment.
• Eye tracking [23],
• Pupil dilation [24],
• Biological parameters measurable by ordinary smart watches (e.g. pulse oximetry [25], IMU [26]).
4. Conclusion and Future Work
In this paper, we have proposed a cognitive architecture which can trivially be implemented for a real-life use-case.
The first level of innovation we have added is by replacing the cognitive layers of the human cortical model with a
natural-human. To enable this, we have added a perception-centric block in the architecture. The overall architecture,
called PeC-HiCA, is trivial to implement. At the same time by having a natural-human in the loop it is deemed to be
more acceptable by industries who want to rapidly use a cognitive architecture. At the same time having a human in
loop also makes it more acceptable by the upcoming AI regulations. Lastly, the perception-centric block also adds a
novel attribute to the existing AI architectures. Current generation of AI algorithms do not incorporate the perception
of the users as an integral part of the training and operation of the algorithm. To our knowledge, this is the first time
user-perception is an integral part of the algorithm.
We are looking for industrial partners who would be willing to co-develop this architecture for a use-case and
demonstrate the merits of a fully cognitive AI automation.
References
[1] G. F. Striedter and R. G. Northcutt, Brains through time: a natural history of vertebrates. Oxford University Press, 2019.
[2] J. M. Fuster, “Upper processing stages of the perception–action cycle,” Trends in cognitive sciences, vol. 8, no. 4, pp. 143–145, 2004.
[3] K. D. Harris and G. M. Shepherd, “The neocortical circuit: themes and variations,” Nature neuroscience, vol. 18, no. 2, pp. 170–181, 2015.
6
Amit Kumar Mishra / Procedia Computer Science 00 (2022) 000–000
[4] A. M. Thomson, “Neocortical layer 6, a review,” Frontiers in neuroanatomy, vol. 4, p. 13, 2010.
[5] B. Schuman, S. Dellal, A. Prönneke, R. Machold, and B. Rudy, “Neocortical layer 1: an elegant solution to top-down and bottom-up integration,” Annual review of neuroscience, vol. 44, pp. 221–252, 2021.
[6] J. Son and A. K. Mishra, “A survey of brain inspired technologies for engineering,” in 2016 Pattern Recognition Association of South Africa
and Robotics and Mechatronics International Conference (PRASA-RobMech). IEEE, 2016, pp. 1–6.
[7] R. Penrose, E. Severino, F. Scardigli, I. Testoni, G. Vitiello, G. M. D’Ariano, and F. Faggin, “Artificial intelligence versus natural intelligence,”
2022.
[8] J. Son and A. Mishra, “Exgate: Externally controlled gating for feature-based attention in artificial neural networks,” arXiv preprint
arXiv:1811.03403, 2018.
[9] M. Valipour, B. You, M. Panju, and A. Ghodsi, “Symbolicgpt: A generative transformer model for symbolic regression,” arXiv preprint
arXiv:2106.14131, 2021.
[10] A. Holzinger, M. Plass, M. Kickmeier-Rust, K. Holzinger, G. C. Crişan, C.-M. Pintea, and V. Palade, “Interactive machine learning: experimental evidence for the human in the algorithmic loop,” Applied Intelligence, vol. 49, no. 7, pp. 2401–2414, 2019.
[11] R. M. Monarch, Human-in-the-Loop Machine Learning: Active learning and annotation for human-centered AI. Simon and Schuster, 2021.
[12] D. Ramsauer, M. Dorfmann, H. Tellioğlu, and W. Kastner, “Human perception and building automation systems,” Energies, vol. 15, no. 5, p.
1745, 2022.
[13] V. Ewald, R. S. Venkat, A. Asokkumar, R. Benedictus, C. Boller, and R. M. Groves, “Perception modelling by invariant representation of deep
learning for automated structural diagnostic in aircraft maintenance: A study case using deepshm,” Mechanical Systems and Signal Processing,
vol. 165, p. 108153, 2022.
[14] S. Haykin, Y. Xue, and P. Setoodeh, “Cognitive radar: Step toward bridging the gap between neuroscience and engineering,” Proceedings of
the IEEE, vol. 100, no. 11, pp. 3102–3130, 2012.
[15] S. Haykin, “Artificial intelligence communicates with cognitive dynamic system for cybersecurity,” IEEE Transactions on Cognitive Communications and Networking, vol. 5, no. 3, pp. 463–475, 2019.
[16] “Regulatory framework proposal on artificial intelligence,” https://digital-strategy.ec.europa.eu/en/policies/regulatory-framework-ai, note =
Accessed: 2022-09-05.
[17] C. R. Sims, “Efficient coding explains the universal law of generalization in human perception,” Science, vol. 360, no. 6389, pp. 652–656,
2018.
[18] T. Brosch, G. Pourtois, and D. Sander, The perception and categorisation of emotional stimuli: A review. Psychology Press, 2010.
[19] I. Mujan, A. S. Anelković, V. Munćan, M. Kljajić, and D. Ružić, “Influence of indoor environmental quality on human health and productivity-a
review,” Journal of cleaner production, vol. 217, pp. 646–657, 2019.
[20] C. D. Nwankwo, A. O. Arewa, S. C. Theophilus, and V. N. Esenowo, “Analysis of accidents caused by human factors in the oil and gas industry
using the hfacs-ogi framework,” International journal of occupational safety and ergonomics, vol. 28, no. 3, pp. 1642–1654, 2022.
[21] A. K. Mishra, “A dikw architecture for cognitive engineering,” Procedia computer science, vol. 123, pp. 285–289, 2018.
[22] I. Pigliautile, S. Casaccia, N. Morresi, M. Arnesano, A. L. Pisello, and G. M. Revel, “Assessing occupants’ personal attributes in relation to
human perception of environmental comfort: Measurement procedure and data analysis,” Building and Environment, vol. 177, p. 106901, 2020.
[23] S. A. Bhoir, S. Hasanzadeh, B. Esmaeili, M. D. Dodd, and M. S. Fardhosseini, “Measuring construction workers’ attention using eye-tracking
technology,” in Proc., ICSC15: The Canadian Society for Civil Engineering 5th Int./11th Construction Specialty Conf. Univ. of British
Columbia, Vancouver, Canada, 2015.
[24] Y.-M. Jang, R. Mallipeddi, S. Lee, H.-W. Kwak, and M. Lee, “Human intention recognition based on eyeball movement pattern and pupil size
variation,” Neurocomputing, vol. 128, pp. 421–432, 2014.
[25] R. A. Stevenson, J. J. Schlesinger, and M. T. Wallace, “Effects of divided attention and operating room noise on perception of pulse oximeter
pitch changes: a laboratory study,” Anesthesiology, vol. 118, no. 2, pp. 376–381, 2013.
[26] W. Tao, H. Chen, M. Moniruzzaman, M. C. Leu, Z. Yi, and R. Qin, “Attention-based sensor fusion for human activity recognition using imu
signals,” arXiv preprint arXiv:2112.11224, 2021.