An Art Analysis by Mistral Pixtral 12B

Introduction

Previous studies of the artistic abilities of VLM Multimodal models such as Microsoft Florence-2-base, Alibaba Cloud Qwen2-VL-2B and DeepSeek Janus-1.3B, have showed that while the results obtained by these models to analyze and explain artworks could be successful, there were still some mistakes. These were caused by the models difficulties to contextualize and understand the artworks or to their tendencies to hallucinate. For instance, the results of the art analysis by Qwen2-VL-2B went as far as seeing objects and figures that were not present, also while correctly identifying the name of paintings and artists such as Vincent Van Gogh’s The Starry Night or Claude Monet’s Impression, Sunrise.

Pixtral 12B has been released on September 17, 2024, as the first-ever Mistral multimodal model under the Apache 2.0 License. With a 400M parameter vision encoder trained from scratch and a 12B parameter multimodal decoder based on Mistral Nemo, Pixtral supports multiple images in the long context window of 128k tokens. As stated on mistral.ai website:

“Pixtral is trained to understand both natural images and documents, achieving 52.5% on the MMMU reasoning benchmark, surpassing a number of larger models. The model shows strong abilities in tasks such as chart and figure understanding, document question answering, multimodal reasoning and instruction following. Pixtral is able to ingest images at their natural resolution and aspect ratio, giving the user flexibility on the number of tokens used to process an image. Pixtral is also able to process any number of images in its long context window of 128K tokens. Unlike previous open-source models, Pixtral does not compromise on text benchmark performance to excel in multimodal tasks”.

The aim of this new study is to see the results obtained when processing artworks with Pixtral 12B — to evaluate its abilities, as well as compare them with the results obtained with the other models, as all the artworks used for this study are exactly the same which have been used in our previous studies with other VLM models.

Analysis of Raphael’s The School of Athens, 1509-1511, Apostolic Palace, Vatican City

We have processed Raphael’s The School of Athens with Pixtral 12B.

Here is the results that were delivered with the prompt “Explain the image”, without adding any information, context or elements related to the fresco or its artist:

“The image depicts a detailed fresco titled "The School of Athens," created by the renowned Italian artist Raphael. This masterpiece is part of the Stanze di Raffaello (Raphael's Rooms) in the Apostolic Palace in the Vatican. Key elements of the fresco include:

1. Central Figures: At the center of the composition, two prominent figures are walking and conversing. These are traditionally identified as Plato (left, holding his hand upward) and Aristotle (right, with his hand extended horizontally). They are surrounded by other notable philosophers and scholars.
2. Architectural Setting: The scene is set in an architectural framework reminiscent of classical Greek and Roman styles, with columns, arches, and detailed ornamentation. The background features a large archway leading to another building, adding depth to the composition.
3. Surrounding Figures: Numerous figures are depicted engaged in various activities such as reading, writing, and conversing. They are dressed in classical attire, emphasizing the theme of ancient philosophy and learning.
4. Sculptures and Decorations: The lower part of the fresco includes decorative elements and statues, adding to the grandeur and classical aesthetic of the scene.
5. Perspective: The use of perspective creates a sense of depth and three-dimensionality, drawing the viewer's eye into the scene. The figures are arranged in a way that guides the viewer's gaze through the painting. "The School of Athens" is a celebration of classical philosophy and learning, embodying the Renaissance ideals of humanism and the pursuit of knowledge. It is considered one of the most important works of High Renaissance art”.

Pixtral’s analysis of The School of Athens shows an incredible accuracy, especially as compared to Florence-2-base (which confused figures and statues), but also Qwen2-VL-2B which had difficulties to depict the precise activity shown in the fresco. Even if Janus-1.3B, had been able to define the painting must have been made by a Renaissance artist, Pixtral was even able to correctly identify the exact name of the artwork and of its artist. The description, which is very wordy and uses the common vocabulary found in art history descriptions, is clear, precise and highly accurate, recognizing the names of the philosophers, identifying their activity (“emphasizing the theme of ancient philosophy and learning”), as well as fully analyzing the use of perspective.

The analysis of Wassily Kandinsky’s Pastorale (1911), Solomon R. Guggenheim Museum, New York

The second artwork we have prompted to “Explain the image” with Pixtral is an abstract artwork by Kandinsky. The results obtained were very intriguing…

The first batch of results delivered by Pixtral were the following:

“The image appears to be a modern, abstract interpretation of a classical painting. The original artwork being referenced seems to be Raphael's "The School of Athens," as evidenced by the architectural elements and the grouping of figures in the lower part of the image. However, this version has been heavily abstracted and overlaid with vibrant, swirling colors. Key features of the image include:

1. Colorful Overlay: The painting is dominated by a bright, abstract overlay of colors, including yellows, greens, blues, pinks, and other hues. These colors create a dynamic and energetic visual effect that obscures much of the original detail.
2. Architectural elements: Despite the abstraction, remnants of classical architectural elements, such as arches and columns, are still visibile, hinting at the original scene’s setting.
3. Figures: The lower part of the image retains some recognizable human figures, though they are partially obscured by the colorful overlay. These figures seem to be engaged in various activities, consistent with the original painting's theme of philosophical discourse and learning.
4. Composition: The overall composition is circular, with the colorful abstraction creating a sense of movement and energy that radiates outward from the center. This abstract interpretation likely aims to juxtapose the classical themes of learning and philosophy with a modern, expressive style, creating a visually striking and thought-provoking piece.”

The whole description of Kandinsky’s Pastorale seems to have been based (hence, biased) on the previous artwork processed by the model. From this hallucinatory setting, suggesting Kandinsky has made this artwork as an interpretation of The School of Athens, Pixtral unwraps a list of references, mirroring elements between the two artworks and a comparison between the original version and its “abstract” counterpart. Indeed, Kandinsky’s abstract representation of a group of figure in a natural landscape has been misinterpreted by Pixtral as a “reinterpretation” of the classical architectural elements, indicating there are “arches” and “columns” in the Kandinsky’s painting. The model goes as far as interpreting the group of characters as “figures … engaged in various activities, consistent with the original painting's theme of philosophical discourse and learning”. While the aesthetical description is more accurate, in its analysis of colors, Pixtral pushes its theory the artwork is a new version of Raphael’s Renaissance fresco, considering it “aims to juxtapose the classical themes of learning and philosophy with a modern, expressive style, creating a visually striking and thought-provoking piece”.

As the results seems to have been less accurate than expected with the influence of Pixtral’s previous analysis of The School of Athens, we have reloaded the page and processed the same painting a second time — wondering if the interpretation of the model would be the same.

Here is the second description by Pixtral for the Pastorale:

“The image is an abstract painting characterized by bold, vibrant colors and dynamic brushstrokes. The composition features a variety of shapes and forms, including rounded, organic forms and more angular, geometric elements. The color palette is dominated by bright hues such as yellow, blue, pink, green, and red, which are used to create a sense of energy and movement. The painting appears to have a central focus on abstracted natural forms, possibly representing flowers or foliage, with some areas suggesting petals or leaves. The use of overlapping shapes and contrasting colors gives the artwork a sense of depth and complexity. The overall style and technique suggest influences from Expressionism or Fauvism, known for their emphasis on emotional impact and vivid color. The painting appears to have a central focus on abstracted natural forms, possibly representing flowers or foliage, with some areas suggesting petals or leaves. The use of overlapping shapes and contrasting colors gives the artwork a sense of depth and complexity. The overall style and technique suggest influences from Expressionism or Fauvism, known for their emphasis on emotional impact and vivid color. The signature "Dubuffet" is visible in the lower right corner, indicating that the artist is Jean Dubuffet, a notable French artist known for his raw, unrefined style and his contributions to the Art Brut movement”.

The second result is aesthetically accurate, as Pixtral did not misinterpreted the painting again as a tribute to Raphael. The depiction of natural elements and colors, recognizing flowers amid the objects, is much more correct. That being said, the painting is mistakenly identifying the artist as “Jean Dubuffet”, from the Art Brut movement, which is really not the case. The idea that the style and technique suggest “influences from Expressionism or Fauvism” is not entirely wrong, as Kandinsky’s artwork might have been influenced by these artistic movements.

Analysis of The Lady of the Unicorn tapestry by the Maître d’Anne de Bretagne (aka Jean d’Ypres), circa 1500, Musée de Cluny, Paris

The third artwork we have analyzed with Pixtral is a medieval tapestry we have already processed with the other models in our previous studies.

Here is the result obtained with the “Explain the image” prompt:

“The image depicts a detailed and ornate tapestry, which is a form of textile art often used for decorative purposes. The tapestry features a richly colored and intricate design with multiple elements and figures. Key features of the tapestry include: Central Figures: In the center, there are several human figures, possibly representing historical or mythological characters. They are dressed in elaborate costumes, suggesting a scene from a significant event or story. Animals: Surrounding the central figures are various animals, including what appear to be lions and other mythical creatures. These animals are depicted in a stylized manner, contributing to the overall fantastical atmosphere of the tapestry. Background and Patterns: The background is filled with intricate patterns and designs, including floral motifs and possibly heraldic symbols. The use of red and blue colors dominates the background, creating a vibrant and dynamic setting. Flags and Banners: There are flags and banners present in the scene, some of which have inscriptions or emblems. These elements add to the sense of a ceremonial or celebratory event”.

As compared to the results obtained with Florence-2-base (which contained several inaccuracies, including the impossibility to detect the presence of the unicorn even after several attempts) and Qwen2-VL-2B and Janus-1.3B more overall correct descriptions, the analysis given by Pixtral is concise yet accurate. The model successfully identified the presence of several figures, including “mythological characters”. Pixtral’s aesthetical description is correct and precise, describing the heraldic symbols, floral motifs and inscriptions and emblems. The interpretation of a “ceremonial or celebratory event” is not exactly true but it is understandable which elements have led the model on this theory. The model has noticed the presence of mythological, fantastical elements but it did not indicate the presence of the unicorn in the artwork, even if it did recognize the figure of the lion. The historical period is not defined by Pixtral.

Analysis of Claude Monet's Luncheon on the grass, 1865-1866, Musée d’Orsay, Paris

In another attempt to evaluate Pixtral’s art analysis abiltiies, we have tasked the model to process and explain Claude Monet’s Luncheon on the grass from 1865-1866.

Here is the description offered by Pixtral:

“The image is an abstract painting that appears to depict a forest scene. The painting is characterized by its use of vibrant colors and dynamic brushstrokes, which create a sense of depth and movement. Key features of the painting include:

1. Forest Elements: The painting prominently features trees and foliage, represented by various shades of green. The trees are depicted with thick, textured brushstrokes, giving the scene a sense of density and naturalism.
2. Human Figures: There are several human figures integrated into the forest scene. These figures are stylized and abstract, with simplified forms and minimal details. They seem to be engaged in various activities, possibly representing a communal or ritualistic event.
3. Color Palette: The painting uses a rich palette of greens, browns, and other earth tones, along with splashes of brighter colors such as reds and blues. These colors contribute to the overall vibrancy and energy of the piece.
4. Composition: The composition is balanced, with the forest elements and human figures distributed across the canvas. The use of color and form creates a sense of depth and perspective, drawing the viewer's eye into the scene.
5. Style: The painting's style is reminiscent of Expressionism or Abstract Expressionism, characterized by its emphasis on emotional and subjective interpretation rather than realistic representation. The thick, visible brushstrokes and bold colors are typical of these movements. Overall, the painting captures the essence of a forest scene with a focus on the interplay between nature and human activity, using abstract and expressive techniques to convey its themes”.

Let’s journey back: when the same painting had been processed by Florence-2-base, it had made relatively few object detections, mistaking the sitting figures for objects and the tablecloth and food on the floor with a table. With Qwen2-VL-2B, it had correctly defined the right time period (late 19th Century, as based on the clothes worn) as well as the setting, but the model did have some hallucinatory moments — for example, indicating the presence of a “child” nowhere to be seen on the actual painting. Janus-1.3B’s description of the artwork had been similar — more or less accurate and with the correct period found, but once more with the indication of a “child” among the group of figures, and with an interpretation of the artwork as made to look “highly realistic” — not really fitting with Monet’s painting style. In comparison, Pixtral’s aesthetical description is accurate but contains some mistakes. The color analysis made by Pixtral is well done, as well as the correct description of the forest setting and of the human figures. The interpretation of a style characterized by “its emphasis on emotional and subjective interpretation rather than realistic representation. The thick, visible brushstrokes and bold colors are typical of these movements” and conclusion the “painting captures the essence of a forest scene with a focus on the interplay between nature and human activity, using abstract and expressive techniques to convey its themes” show Pixtral’s abilities at analyzing painting style, brushstrokes and artistic content in general. On the other hand, the fact the model interprets the artwork as “reminiscent of Expressionism or Abstract Expressionism” — artistic movements which would not even start until decades later — indicate some mistakes are still made in Pixtral’s art analysis.

Analysis of Wassily Kandinsky's Impression III (Concert), 1911, Lenbachhaus, Munich

We have processed another abstract artwork by Wassily Kandinsky with Pixtral, for the purposes of this study.

In this case, the artwork was Impression III (Concert), which have been already analyzed by Florence-2-base and Janus-1.3B. The results obtained with both these two other models shows the difficulties for VLM models as for now to analyze abstract artistic content. If they both had made an attempt at identifying a subject in this artwork, the results had been for the least inaccurate — with the abstract figures interpreted as flowers in some cases.

Here are the results obtained with Pixtral:

““The image is an abstract painting characterized by bold, vibrant colors and dynamic brushstrokes. The dominant background color is a bright yellow, which serves as a backdrop for various abstract shapes and forms. Key features of the painting include: Color Palette: The painting uses a vivid palette with prominent yellow, red, blue, and black colors. These colors are applied in thick, visible brushstrokes, giving the painting a textured and energetic appearance. Shapes and Forms: The painting contains several abstract shapes and forms, including: A large, dark green or black area in the upper right corner. A red and yellow area in the upper left corner. Blue and black lines and shapes in the center. A series of curved, black lines and shapes that resemble stylized figures or symbols. Composition: The composition is balanced but dynamic, with the various elements distributed across the canvas. The use of color and form creates a sense of movement and energy. Signature: In the bottom right corner, the painting is signed "Kandinsky 1911," indicating that it is a work by the renowned abstract artist Wassily Kandinsky, created in the year 1911. Overall, the painting exemplifies Kandinsky's early abstract style, characterized by its expressive use of color and form to convey emotion and movement”.

Pixtral’s analysis of Kandinsky’s Impression III (Concert) was clearly the most successful. The model was able to identify the name of the artist, as well as the date of the painting from analyzing and correctly transcribing the signature — both Florence-2-base and Janus-1.3B had failed in identifying Kandinsky as the artist for this precise example. The aesthetical description of the shapes, form is very accurate. The model tried to deliver its interpretation of the painting and contextualize it in Kandinsky’s career, indicating it “exemplifies Kandinsky's early abstract style, characterized by its expressive use of color and form to convey emotion and movement”. Interestingly enough, while its color theory is well thought of, Pixtral did not offer a real concrete interpretation of the abstract elements in the scene.

Analysis of The Starry Night by Vincent Van Gogh, 1889, MoMa, New York

The last painting we have processed for this art study with Pixtral is Vincent Van Gogh’s iconic The Starry Night.

The painting had already been processed with all the three models used for our previous studies. While Qwen2-VL-2B and Janus-1.3B had successfully identified the name of the painting and of its artist, there were some remaining inaccuracies — due to Van Gogh’s almost abstract style of representation which might have made it harder to understand. In light of all this, the results were positive. If there were some hallucinations (Janus-1.3B interpreted the village represented as the homeland of Van Gogh), the aesthetic and object descriptions were correct, especially for Qwen2-VL-2B.

Here is the result obtained with Pixtral:

“The image is an abstract painting characterized by its vibrant colors and dynamic brushstrokes. The painting appears to depict a natural scene, possibly a landscape, with a focus on the interplay of light and color. Key features of the painting include:

1. Color Palette: The painting uses a rich palette dominated by shades of blue, green, and yellow. These colors are applied in thick, visible brushstrokes, giving the painting a textured and energetic appearance.
2. Light and Movement: The use of swirling and curved lines creates a sense of movement and energy, suggesting the dynamic nature of light and atmosphere in a landscape. The brushstrokes seem to capture the fluidity and vibrancy of natural elements.
3. Composition: The composition is balanced, with the various elements distributed across the canvas. The painting features a central area with dense, overlapping brushstrokes, possibly representing trees or foliage, and a more open, lighter area to the right, which could symbolize the sky or a body of water.
4. Style: The painting's style is reminiscent of Expressionism or Abstract Expressionism, characterized by its emphasis on emotional and subjective interpretation rather than realistic representation. The thick, visible brushstrokes and bold colors are typical of these movements.
5. Signature: In the bottom right corner, the painting is signed "Kandinsky 1911," indicating that it is a work by the renowned abstract artist Wassily Kandinsky, created in the year 1911 Overall, the painting captures the essence of a natural landscape with a focus on the expressive and dynamic qualities of light and color, using abstract and energetic techniques to convey its themes”.

Pixtral having processed Van Gogh’s The Starry Night right after its analysis of Kandinsky 1911 painting seems to have led to comprehension issues. In fact, the description of the elements in a “natural scene” such trees, foliage, sky etc. is correct, as well as the interpretation of the brushstrokes and colors. But the hallucination begins the moment where Pixtral evokes the painting is once again signed by Kandinsky in 1911, as indicated in the signature in the bottom right (?). Whether the analysis was done too quickly after the previous one or the model got mixed up in its images and descriptions is not clear. But the interpretation it finds as — once again — a reminiscence of “Expressionism or Abstract Expressionism” (which seems to be two of Pixtral’s favorite artistic movements) is wrong in any case.

We have ran again the same prompt to “Explain the image” after having re-started the system to see if the results would be different.

Indeed, here is the description produced by Pixtral for this second round:

“The image is a painting titled "Starry Night" by Vincent van Gogh. The artwork is one of his most famous pieces and is known for its swirling, dynamic sky filled with stars and a crescent moon. The sky is dominated by vibrant blues, yellows, and whites, creating a sense of movement and energy. Below the sky, there is a dark, tall cypress tree on the left side, which contrasts sharply with the bright sky. In the lower part of the painting, there is a small village with houses and a church with a tall steeple. The village is depicted with a series of rooftops and a few trees, giving a sense of a quiet, peaceful settlement. The overall composition and use of color create a sense of harmony and balance, making "Starry Night" a timeless masterpiece of Post-Impressionist art”.

If the description is much shorter, it is now completely accurate and much more precise. The name of the artist has been correctly identified by Pixtral, as well as the title and the description of all the elements in the artwork. The interpretation of the painting as “a timeless masterpiece of Post-Impressionist art” shows Pixtral’s highly good abilities to analyze and process art — and is also much more accurate.

Conclusion

To conclude this study, we can attest that Pixtral 12 B definitely has a high potential to process, analyze and describe works of art. The main issue during the analysis was the risk of hallucinations, which happened to be higher when two artworks were analyzed in a very short time period, one right after the other. In this case, the model seems to get highly confused between the two images and descriptions and either incorporate the wrong interpretation (for example, miscomprehending Kandinsky’s Pastorale for an abstract version of Raphael’s The School of Athens) or mis-identify one artwork for the other — for instance, when it stated The Starry Night was painted by Kandinsky in 1911 right after analyzing a painting by Kandinsky from this date.

Pixtral 12B, much like Florence-2-base, Qwen2-VL-2B and Janus-1.3B, has tendencies to miscomprehend based and biased from its previous knowledge, leading to hallucinations: for example when processing an abstract artwork, it indicated it was made by “Jean Dubuffet” for no clear reason whatsoever.

The art history vocabulary used and the aesthetical and color descriptions by Pixtral were extremely accurate overall.

All in all, Pixtral 12B has a high potential including for art analysis and it would be interesting to see how its abilities fare to analyze elements of text in images as well in a future study — also to compare the results with those of other models for the same tasks.

Bibliography

1. Maître d’Anne de Bretagne (aka Jean d’Ypres), The Lady and The Unicorn (circa 1500), Musée de Cluny, Paris
2. Raphael (1483-1520), The School of Athens (1509-1511), Apostolic Palace, Vatican City
3. Claude Monet (1840-1926), Luncheon on the grass (1865-1866), Musée d’Orsay, Paris
4. Vincent Van Gogh (1853-1890), The Starry Night (1889), MoMa, New York
5. Wassily Kandinsky (1866-1944), Impression III (Concert) (1911), Lenbachhaus, Munich
6. Wassily Kandinsky (1866-1944), Pastorale (1911), Solomon R. Guggenheim Museum, New York