Labelling the psychological impact of colour in films for deep learning processing based on the humanities approach: RGB information in colour classification in film

SUMMARY Inspired by Goethe’s theory of colour and his aversion to Newton’s scientific theories, we performed an experiment to investigate the psychological impact of colour in film. We converted 550 films into sequences of images and then processed them to extract the RGB information within. We then compared this data and matched them to classifications of colour roles in terms of their psychological influence on film viewers. For this we consulted Bellantoni, Patti (2005)’s book If It’s Purple, Someone’s Gonna Die: The Power Of Colour In Visual Storytelling , which is a guide for filmmakers to select the appropriate colours for their films and also describes the psychological and emotional effects of certain colours in film on viewers. Finally we extracted RGB information from our image sequence archives to match them to classifications of the role of colour and their psychological influences on film viewers. In spite of the fact that our results are subject to certain limitations such as the subjectivity of our researchers, we believe our findings from the experiment have made contributions to the development of AI for use in film colour classification with regard to their psychological impacts on viewers. In future endeavors, we expect to include additional information, such as shapes, tints and so on. Thus, our methodology and materials may produce even better results if supplementary resources were to be available.

Labelling the psychological impact of colour in films for deep learning processing based on the humanities approach: RGB information in colour classification in film SUMMARY Inspired by Goethe's theory of colour and his aversion to Newton's scientific theories, we performed an experiment to investigate the psychological impact of colour in film. We converted 550 films into sequences of images and then processed them to extract the RGB information within. We then compared this data and matched them to classifications of colour roles in terms of their psychological influence on film viewers. For this we consulted Bellantoni, Patti (2005)'s book If It's Purple, Someone's Gonna Die: The Power Of Colour In Visual Storytelling, which is a guide for filmmakers to select the appropriate colours for their films and also describes the psychological and emotional effects of certain colours in film on viewers. Finally we extracted RGB information from our image sequence archives to match them to classifications of the role of colour and their psychological influences on film viewers. In spite of the fact that our results are subject to certain limitations such as the subjectivity of our researchers, we believe our findings from the experiment have made contributions to the development of AI for use in film colour classification with regard to their psychological

Introduction
Colour is a natural part in the way humans perceive and experience the world. The 'natural' includes the psychological and subjective responses of our brains in the process whereby our eyes respond to light. However, Isaac Newton (1643Newton ( -1727 ushered in a scientific revolution by pioneering a scientific approach to light and colour. For him, colour was a physical manifestation, existing outside of the body, per his interpretation of the famous experiments where different colours were displayed through a prism. Johann Wolfgang von Goethe (1749Goethe ( -1832 disagreed with Newton and instead espoused for the naturalistic response of humans to colour. He criticized Newton's theory of colour as a scientific and objective decomposition of natural experiences. Goethe (1840) published Zur Farbenlehre (Theory of colours) to repudiate Newton's views and articulated that the Newtonian teachings were false in 1810. His polemic against Newton was a formulation of the psychological and philosophical account in the way we take colour in the tradition of classical German philosophy, known as German Idealism from Kant to Hegel, which mainly dealt with epistemological problems of securing the possibility of objective knowledge in contrast to subjective knowledge. Goethe's understanding of colours was dismissed as fallacious and incorrect by the scientific community. However, his theory of colour still inspired numerous philosophers and artists.
Based on the discussions of Goethe's aversion to Newton's ideas, we started research to develop AI (Artificial Intelligence) applications to classify the psychological impacts of colour in film on viewers. For this purpose, we suggested using AI algorithms to classify the subjective and psychological impacts of colour in films on human beings in our previous research. Drawing on Goethe's colour theory, we developed separate colour categories of films for classifying the psychological responses of viewers to colours in our analysis of La La Land (2016), an American musical romance in our previous paper (Han et al., 2018). Our previous research was limited because we developed a method of colour analysis for classifying the psychological responses of viewers to colours based on a single film. We also recognized that further research based on more film analysis and audience surveys would have to be completed.
Against this backdrop, we launched a subsequent, long-term, multi-stage research project wherein the psychological impact of colour on film viewers would be used as labels in a deep learning process based on the analysis of massive amounts of image sequences from 550 films. Currently, we have reached a significant milestone in our overall research goals. We finished converting 550 films to image-sequence archives. Based on this milestone, we propose a system that could match RGB information to classifications of colour roles in their psychological influences on film viewers. We want to articulate that our approach to colour classification comes from a humanities background but with an AI engineering approach. The main goal of our research is to simply suggest AI applications based on theories of mental imagery in the humanities for the long term. Our team is composed of independent researchers and students from the disciplines of the humanities and social sciences. Furthermore, our research is subject to budgetary constraints which preclude us from using resource intensive approaches. At the current stage of this research, we do not intend to develop a fully-fledged AI application as this will require further experimentation in multiple steps involving technical experts in the field of AI. Current research aims to galvanize the humanities into initiating further interdisciplinary research of this nature.

Background Colour classification schemes
Colour classification schemes were introduced over the last three hundred years in colour theory (Graumann, 2007). From the time of Aristotle, the form of light was thought to be white. The modern and scientific view of colour arose with Isaac Newton (1643-1727. He pioneered a modern understanding of optics, light and colour and dismantled the conventional view on light. In the late 1660s, he performed a string of experiments using prisms to deduct the nature of light and colour. In his experiments, white light was decomposed into the different colours through prism refraction: red, orange, yellow, green, blue and violet. Newton concluded from his experiments that any physiologically-based explanation for light and colour was false. For him, colour was not a physiological process in our subjective perceptions as human beings, but a physical and scientific object existing outside of the body. Newton's theory of colour was published in Opticks: or, A Treatise of the Reflexions, Refractions, Inflexions and Colours of Light (Newton, 1730). It embodies the objectivity of the scientific approach in the modern era. However, Newton's discoveries came under fierce criticism from Johann Wolfgang von Goethe (1749Goethe ( -1832. Disagreeing with Newton, he advocated for humans' subjective experiences of colour. He emulated Newton's experiments on colours after returning from his stay in Italy from 1786 to 1788 (Barsan & Merticarius, 2016).
Goethe saw that Newton's experiments on colour could be reproduced in his own experiments. However, he was dissatisfied with Newton's conclusion that colour was merely a physical and scientific phenomenon. He protested against Newton's theory in Zur Farbenlehre (Theory of Colours) in 1810 in which he presented his case against Newton and the formulation of a psychological and philosophical interpretation on the perception of colour. Goethe considered himself more of a scientist than a philosopher and regarded Zur Farbenlehre as his most important work. He ardently promoted his theory of colour as his greatest and ultimate achievement in his interviews with Eckermann. "It was also prejudicial to me that I discovered Newton's theory of light and colour to be an error, and that I had the courage to contradict the universal creed. I discovered light in its purity and truth, and I considered it my duty to fight for it" (Eckerman, 1850). In spite of Goethe's struggle with Newton, his theory of colours was banished from the modern scientific community as erroneous. However, Farbenlehre still inspired numerous philosophers and artists. Among them, Georg Wilhelm Friedrich Hegel (1770Hegel ( -1831 and Arthur Schopenhauer (1799-1721 were philosophers who communicated with Goethe on the nature of colour. Hegel supported Goethe's critique of Newton and his conception of colour is heavily influenced by Goethe's theory of colour (Peters, 2017): "The ineptitude and incorrectness of Newton's observations and experiments complement their inanity, and Goethe has even shown that they are not entirely above board, but one of Newton's most glaring and elementary errors is the false assertion that when a monochromatic part of the spectrum produced by a prism passes through a second prism, it will also reappear in its merely monochromatic form." (Hegel, 1970) Throughout his life, Hegel mainly focused on developing a philosophical theory on consciousness. He contributed to his theory of colour with respect to Goethe's Farbenlehre in a small fraction of his writings on the philosophy of nature and aesthetics. Arthur Schopenhauer, the son of Johanna Schopenhauer (1766-1838), a popular novelist at the time who formed a friendship with Goethe, was directly influenced by Goethe's Farbenlehre. Young Arthur became acquainted with Goethe and his opposition to Newton's Optik. Under the influence of Farbenlehre, he expressed his own theory of colour: Über das Sehn und die Farben (On vision and colours) in 1816 after extensive discussions with Goethe. In his book, he judged Goethe's Farbenlehre positively and showed admiration for his achievements (Mertens, 1997). He drew on the role of the retina in his attempts to make Goethe's theory a more rigorous subjective system and claimed that the retina was stimulated by opposing poles of colour. The thrust of his theory was that "colour is the qualitative division of the activity of the retina" (Schopenhauer, 2010). However Schopenhauer's theory of colour did not appeal to Goethe and he distanced himself from Schopenhauer after his publication of Über das Sehn und die Farben. Eva Heller had recently done research on the connection between colour preferences and the psycho-physiological reliance on statistical research (Heller, 2004). She conducted a survey on 2,000 people about their psychological responses to colours in relation to 200 terms on different topics such as culture, politics, emotions, etc. Heller's findings could be regarded as statistical proof on earlier findings in psychology on colour. Her result on the correlation between the psychological and symbolic meaning of colour was published in her book Wie Farben wirken: Farbpsychologie as summarized in Table  Heller's work shows that people associate certain psycho-physiological sensations with a symbolic meaning of colour. In certain media, such as advertisements and film, colour evokes mental images in association with psychological imagination, lexical and empirical knowledge in a social and cultural context.

Colours in films
The film industry saw the rise of colour-process films in the 1930s but most films were still photographed in black and white film until the 1960s. The advent of colour television galvanized the industry into rapidly shifting to colour-processed film. There were many experiments on the use of colour for symbolic purposes in the film industry. In particular, colour has been used to create subconscious elements in films. Colour has usually been utilized in the emotional, expressive and atmospheric rather than any intellectual context. "Psychologists have discovered that most people actively attempt to interpret the lines of a composition, but they tend to accept colour passively, permitting it to suggest moods rather than objects" (Giannetti, 2014).

Bellantoni's book If It's Purple, Someone's Gonna Die: The Power of Colour in Visual
Storytelling is a filmmakers' guide in selecting the right colours for their films. It describes the psychological and emotional effects of certain colours in film on the audience. Based on her twenty-five years of research on the effects of colour on behavior, Bellantoni categorized six major colours in order to understand their influences in film. For example, she explained that some films are predominantly influenced by specific colours in order to represent certain themes and characters.

Colour Classification in Artificial Intelligence
Currently, machine learning (ML) has become a dominant problem-solving technique in artificial intelligence (AI). Tom Mitchell is an American computer scientist who contributed to writing the first standard textbook on machine learning in which he defined machine learning as follows: "A computer program is said to learn from experience E with respect to some class of tasks T and performance measure P, if its performance at tasks in T, as measured by P, improves with the experience E" (Mitchell, 1997). In Mitchell's writings, T, E, and P are defined as: • task (T), either one or more, • experience (E) • and performance (P).
Hence, machine learning consists of a set of tasks and the experience leading to performance running.
Machine learning algorithms can be divided into two learning types: supervised and unsupervised learning. Supervised learning refers to algorithms being trained with a set of labeled training data. The computer is provided with labeled training data as correct answers. The machines are then trained with this labeled data. Hence, in supervised learning, there is training data where we have an input object and an output object. In unsupervised learning the algorithm is simply provided with a load of data without any correct answers (or labels) to guide it. The machines then aim to find any hidden patterns through the learning process (Bell, 2015).
The criteria for colour classification in our research require extensive experiments, including the recruitment of participants. However, at the current stage of our research and due to budgetary constraints, we were unable to perform the desired experiments which is why we decided to deploy a more simplistic methodology, where instead of using a larger sample (such as an entire audience), the researchers themselves picked out the representative images for each psychological colour classification as our training dataset. We will be looking to perform more sophisticated and elaborate experiments in the next stages of our research.
The datasets of our model include the 'Psychological Impact" of colour as the labeled variable as per Bellantoni's classification of colour's psychological impacts; and 'RGB value' as the input variable. Our ML model is expected to use the RGB values from the carefully selected scene images from films to be trained for the matching colour and psychological impacts. Hence, the machine learning algorithm will be trained to map the extracted RGB values to the psychological colour classifications.
RGB is an acronym that stands for the three primary colours -red, green and blue, which, as radiated light, can be mixed to produce any other colour. A combination of the three produces white light by the additive process. RGB is used for computer displays while the CMYK system is used to provide the hard copy (Paterson, 2013).

Materials and methods
Our experiment consists of three components. Firstly, we constructed an image sequence archive from 550 films as the dataset. Secondly, we personally selected 20 image sequences from the archive as a representative sample for each of Bellantoni's psychological colour classifications. Finally, we extracted the RGB data from this sample of representative images, to be matched with the psychological colour classifications. This would serve as the training data-set for the machine learning algorithm.
As for the films, we turned to two film-reference books: 1001 Movies You Must See Before You Die (Schneider & Smith, 2017) and Film-Klassiker (Töteberg, 2016), as well as film lists published by media organisations such as the BBC's 100 Greatest Films of the 21st Century (BBC, 2016) and All-Time 100 movies (Time, 2005). Moreover, we turned to film awards lists from the Academy Awards and the Cannes Film Festival. We then selected 550 films from these lists that were available in video file formats. We adopted the MP4 (MPEG-4) and MKV video file formats. MP4 is a digital multimedia container format used to store video and audio and the MKV is a file format that can hold video, audio, picture, or subtitle tracks all in a single file.
We converted the video files of all 550 films into image-sequence archives. We  Secondly, we selected 20 image sequences from the image sequence archive to be a representative sample for each of Bellantoni's classification of the psychological impact of colour and saved them into folders with the name psychological impact.
Thirdly, we extracted the RGB information from the representative samples of Bellantoni's classifications. This extracted RBG data for the colour classification will be the label which the deep learning algorithm will train on to learn to map colour to psychological impact. As for the computational process, we chose the Python language as the programming environment with libraries including pillow module of PIL (Python Imaging Library version 7.1.2), opencv-python 4.5.3.56 and cvlib 0.2.7 on MacBook Air (M1, 2020) with 8GB memory and macOS Monterey version 12.3).
PIL is a free open-source library for the Python programming language that adds support for opening, manipulating, and saving many different image file formats. Development of the PIL, was discontinued in 2011. Subsequently, a successor project named Pillow forked the PIL repository and replaced the original PIL. OpenCV (Open Source Computer Vision Library) is a library of programming functions for real-time computer vision. The library is cross-platform and free to use under opensource licenses. The opencv-python is a library of Python bindings designed to solve computer vision problems (OpenCV, 2022).
We chose the area of an image sequence and used the OpenCV library to draw out the RGB values from the colours therein. The specific colour chosen from an image frame of multiple colours is picked manually by the researchers based on their subjective and likely psychological impact on the viewer. We retrieved the RGB values in the area of the images with the goal of extracting RGB values that match the colour classifications through sampling of the drawn RGB values. Table 3 summarises the extracted RGB information about the colours and their matching psychological impacts. For example, the blue in Cerebral Blue as used in films gives off to viewers an impression or ambience of the intellectual, and our RGB values of this specific colour are shown in Table 3.

Subjectivity of classification of colour
However, we did not find that the drawn RGB values were significantly differentiated between the different sub-classifications of colours. This is likely due to the small sample size of the research group itself. This also comes down to the inherent subjectivity of colour classification itself, as we classified the psychological impact of colour on viewers based on Bellantoni's book. Moreover, subjectivity is also reflected in the choice of scenes and how colours were chosen and then matched with the Bellantoni's colour classifications. Since we performed the experiment with budgetary constraints, there were certain limitations in the experiment, for example not using a much larger sample size of individuals to pick out colours that were perceived to match with Bellantoni's colour classification. With a commensurate increase in resources, we expect that our materials and methodology will likewise improve, and we could get much better results and differentiation between the classes of the psychological impacts of colour.

Colour Psychological meaning RGB Values
The extracted RGB values shown in Table 3 have multiple and dynamic values. They should be converted to an index or range of values if they are to be used as labels for the machine learning classification algorithm. This process would be far better if the selection of image sequences and classification were performed by a big audience of test candidates. We hope that we can accomplish these experiments in further research.
Also, we could deploy better algorithms in our research if we have more technical experts participating in our research.

Practical implications
Our research produced several practical implications. Firstly, our datasets were composed of massive image sequences from 550 films, and this could motivate further research on developing AI algorithms that could address the cultural and psychological impacts of film images. For example, our research could be used to apply Gaston Bachelard's imagination of matter and Gustav's Jung's theory on images and push for the development of more AI applications in the humanities. In future, we believe our research will contribute to further enabling AI to classify images based on theories from the art and humanities.

Conclusion
In this paper, we propose a foundational framework wherein RGB information can be used to match classifications of the role of colour in terms of their psychological influence on film viewers, as based on Goethe's theory on colour. We constructed massive amounts of image-sequences from 550 films and extracted RGB values matching the colour and its psychological categorisations. Further research could use this framework to develop machine learning algorithms (or use existing ones) to automatically classify film images based on their psychological impacts. Colour also has other properties such as 'hue' and 'saturation' and these can be used as well.
The meaning of colour is varied and paradoxical and our results could be criticised for their subjectivity as we personally picked the images for the classification ourselves. However, we've contributed to devising an AI framework for colour classification that tackles the psychological impacts from images and motion pictures and we believe that our research could be considered a pioneering contribution to the field of AI in the humanities, in contrast to the rapid developments of AI in the scientific and engineering fields. We have come up with an enhanced paradigm for the potential use of AI in the humanities that could encourage further research. In future, our research will look for new opportunities to be performed in more extensive ways. Unatoč činjenici da su naši rezultati podložni određenim ograničenjima poput subjektivnosti naših istraživača, vjerujemo da su rezultati našeg eksperimenta pridonijeli razvoju umjetne inteligencije koja se koristi za klasificiranje boja u filmovima u vidu njihovog psihološkog utjecaja na gledaoce. U budućim pothvatima planiramo uključiti dodatne informacije poput oblika, nijansi i sl. Smatramo kako bi naši materijali i metodologija mogli polučiti bolje rezultate kad bi nam bili dostupni dodatni resursi.