Bongard Problems

Bongard Problems are visual puzzles invented by the Russian computer scientist Mikhail Bongard. In contrast to standardized cognitive tests like IQ tests, they are not mainly about speed or pattern recall, but about forming the right abstraction. Bongard Problems are a great way to test problem-solving skills und creative thinking.

How it works

Each puzzle consists of 12 images: six on the left and six on the right. The left set follows one hidden concept, while the right set violates it. The task is to infer that concept.

Additional Resources

• The On-Line Encyclopedia of Bongard Problems: Link
• Problem collection by Harry Foundalis: Link
• Foundalis' PhD thesis: Link

Examples

Three Bongard problems follow. Try to solve them first, or jump straight to the solutions in the callouts.

• Example 1:

• Example 2:

• Example 3:

Artificial Intelligence

Because each problem has a different underlying rule, Bongard Problems are an interesting benchmark for artificial intelligence.

From (¹):

The key point is that Bongard Problems require inventing concepts, not just classifying known categories. If that is true, then standard supervised machine learning is fundamentally limited on this benchmark.

As each Problem consisists of very few (six) training-examples, traditional computer-vision methods do not work (²):

Infamous for requiring massive amounts of data to perform well at image classification problems, deep learning has so far been unable to solve Bongard problems (BPs)

Tests with ChatGPT

With the prompt "Solve this Bongard Problem" and Example 1 attached, ChatGPT 5.2 correctly finds the solution:

The governing distinction is convexity.

Left set: All shapes are convex — for any two points inside the shape, the line segment between them lies entirely within the > shape. These include regular/irregular polygons, a circle, and an > ellipse, all without indentations.

Right set: All shapes are non-convex (concave) — each has at least one indentation or inward “bite,” meaning there exist two points inside the shape whose connecting segment passes outside > the boundary.

Answer: Left = convex shapes Right = concave (non-convex) shapes

This may be because this exact image was present in training data. Since there are only about 400 classical Bongard Problems, dataset contamination is a serious limitation for this test.

Example 2 was not solved by ChatGPT:

The key distinction is whether boundaries intersect.

Left set: The circles intersect each other — their outlines cross at one or more points (true geometric intersection).

Right set: The circles do not intersect — they are either:

• completely separate, or
• one is entirely inside the other (nested) without touching or crossing.

Answer: Left = intersecting circles Right = non-intersecting circles (disjoint or fully contained)

Obviously, "Left set: The circles intersect each other" is false for the shown puzzle.

Quantitative Analysis

The paper Neural Networks for Abstraction and Reasoning (¹) evaluates several systems on ARC (a benchmark inspired by Bongard-style abstraction tasks):

Reported headline numbers on the public ARC splits:

• DreamCoder (neuro-symbolic): 16.5% on ARC-Easy, 4.5% on ARC-Hard
• GPT-4 (LLM setup from the paper): 21% on ARC-Easy, 8% on ARC-Hard
• Final ensemble (Icecuber + DreamCoder + GPT-4): 57% on ARC-Easy, 40% on ARC-Hard

Firstly, pure neural approaches still do not dominate this benchmark. The strongest single-system performance remains heavily hand-crafted search (Icecuber-style DSL search), while neural systems help only part of the way.

Secondly, model diversity matters. The paper reports only 37% overlap between tasks solved by DreamCoder and GPT-4, meaning they often solve different tasks. Combining them gives a clear gain over either one alone.

Footnotes

1. Bober-Irizar, Mikel, and Soumya Banerjee. 2024. “Neural Networks for Abstraction and Reasoning.” Scientific Reports 14(1):27823. doi:10.1038/s41598-024-73582-7. ↩ ↩²

2. Yun, Xinyu, Tanner Bohn, and Charles Ling. 2020. “A Deeper Look at Bongard Problems.” Pp. 528–39 in Vol. 12109, Lecture Notes in Computer Science, edited by C. Goutte and X. Zhu. Cham: Springer International Publishing. ↩