{"id":39754,"date":"2018-06-17T19:59:19","date_gmt":"2018-06-17T19:59:19","guid":{"rendered":"http:\/\/www.labri.fr\/perso\/barla\/blog\/?p=39754"},"modified":"2019-08-13T16:35:16","modified_gmt":"2019-08-13T16:35:16","slug":"proximal-generative-model","status":"publish","type":"post","link":"https:\/\/www.labri.fr\/perso\/barla\/blog\/?p=39754","title":{"rendered":"Mechanistic visual perception?"},"content":{"rendered":"<p id=\"top\" \/>\n<h3 class=\"wp-block-heading\">In praise of Ecological Optics<\/h3>\n\n\n\n<p>Stimuli are usually generated via interactions of light, shape and matter through rendering equation. But only gives information about 0th-order image structure (pixel intensity and color)! Some studies connect perception directly to physical properties, others to mid-level factors.&nbsp;This raises new questions about how these properties are detected in the observed image!&nbsp;Other studies connect perception directly to image properties (histograms, Fourier, etc).&nbsp;But this does not explain how these properties are related to the real world; when are they relevant?<\/p>\n\n\n\n<p>Images are already structured when they reach the eye, in ways that depend on physical properties.&nbsp;Proposed approach: characterize this structure objectively, then study how it is perceived subjectively.&nbsp;This suggests to rely on a generative model that directly provides proximal information to work on.&nbsp;If we extend this to stereo active vision in motion, then we study the structure of Gibson&#8217;s optic array.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Potential vs effective cues<\/h3>\n\n\n\n<p>Distinguish between the structure potentially available to an observer, and the structure effectively used.&nbsp;Some available image cues might be entirely missed, depending on physiology, attention, experience, etc.&nbsp;Other cues might be available under some conditions but not others, and active search might bring them out.&nbsp;Ex: compressed reflections and texture directions for shape, sharp highlights with good coverage for materials. The question that must be answered is &#8220;What is it <em>in the image&nbsp;<\/em>that the brain&nbsp;<strong>can&nbsp;<\/strong>use to [&#8230;]?&#8221;<\/p>\n\n\n\n<p>Irrespective of particular conditions at a given time and place, image cues are still potentially present.&nbsp;Potential cues are closest to intrinsic physical properties, but are not equivalent: many-to-one relationships!&nbsp;A proximal generative model should first identify potential cues from physical properties, akin to invariants.&nbsp;Then depending on the actual viewing configuration, some of these potential cues should be filtered out.&nbsp;This characterizes what is made available to a visual system (human or other); again, not all might be picked up.&nbsp;Vision might also fill in unavailable cues from past experience\/memory\/inference; but this is out of our scope.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Structure from scratch<\/h3>\n\n\n\n<p>It is hypothesized that vision relies on available information in the structure of the optic array; why not creating it from scratch?&nbsp;This is what is done through painting and drawing: if enough hints to the structure is provided, then perception is possible.&nbsp;It should thus be possible to analyze paintings and drawings in exactly the same way we analyze retinal sensations.&nbsp;One obvious limitation is that motion and visuo-motor feedback are not there, as described by Gibson in his &#8220;depiction&#8221; chapter.<\/p>\n\n\n\n<p>Creating structure from scratch brings new means to experiment with vision: image cues may be distorted or removed surgically.&nbsp;The more we go in this direction, the more we will get away from the real-world and into the mechanisms of visual perception.&nbsp;Pushing this idea to the extreme, one ends up playing with illusions that provide non-veridical information through image structure.&nbsp;This permits testing of the limitations of the visual system itself, even potentially identifying ecological situations in which it could occur!<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Mechanistic visual perception<\/h3>\n\n\n\n<p>On one side, the world provides for visual <strong>information<\/strong>: once a viewpoint is set up, one should be able to express <em>potentially<\/em> available structure in the ambient optic array, and relate it to the optical, geometrical and mechanical properties of objects and substances populating the environment; then once interaction with between lighting and objects, or among objects, are taken into account, one should be able to&nbsp;characterize the&nbsp;<em>effectively&nbsp;<\/em>available structure for the observed scene. This view-dependent process belongs to the field of Ecological Optics.<\/p>\n\n\n\n<p>On the other side, the observer might exhibit different&nbsp;<strong>capacities&nbsp;<\/strong>regarding the perception of available information: each source of information might be subjected to different <em>ambiguities;&nbsp;<\/em>these may require to resort to&nbsp;<em>cue combination&nbsp;<\/em>strategies as well as to&nbsp;<em>inference&nbsp;<\/em>to some extent. These task-dependent aspects are studied using Psycho-physics.<\/p>\n\n\n\n<p>Understanding visual perception requires to explain capacities with respect to available information through the postulation of <strong>mechanisms<\/strong>. They must implement several functions: describe which aspect of the <em>proximal generative model<\/em> are assumed to be picked up by the Human Visual System, predict how the tempering of information leads to&nbsp;<em>systematic alterations&nbsp;<\/em>of perceptual outcomes through image manipulations and creations&nbsp;(e.g., illusions, drawings) ; suggest which part(s) of the brain might be involved in&nbsp;<em>cue processing&nbsp;<\/em>and how!<\/p>\n\n\n\n<div class=\"wp-block-image\"><figure class=\"aligncenter\"><a href=\"https:\/\/www.labri.fr\/perso\/barla\/blog\/wp-content\/uploads\/2018\/06\/MechanisticVisualPerception.png\"><img loading=\"lazy\" decoding=\"async\" width=\"972\" height=\"447\" src=\"https:\/\/www.labri.fr\/perso\/barla\/blog\/wp-content\/uploads\/2018\/06\/MechanisticVisualPerception.png\" alt=\"\" class=\"wp-image-39845\" srcset=\"https:\/\/www.labri.fr\/perso\/barla\/blog\/wp-content\/uploads\/2018\/06\/MechanisticVisualPerception.png 972w, https:\/\/www.labri.fr\/perso\/barla\/blog\/wp-content\/uploads\/2018\/06\/MechanisticVisualPerception-300x138.png 300w, https:\/\/www.labri.fr\/perso\/barla\/blog\/wp-content\/uploads\/2018\/06\/MechanisticVisualPerception-768x353.png 768w\" sizes=\"auto, (max-width: 972px) 100vw, 972px\" \/><\/a><\/figure><\/div>\n\n\n\n<h3 class=\"wp-block-heading\">Multi-modal perception<\/h3>\n\n\n\n<p>Touch is a more active sensory modality. Hence mostly everything is about turning potential into effective cues. This does not mean that all effectivelly available cues will be picked up. Blind subjects might be especially expert in this respect, as well as cloth designers. Both might not only use relevant active tactile strategies, but also have a refined perceptual distinction.<\/p>\n\n\n\n<p>Vision and touch might share the same potential cues, or at least their relevant scales are likely to overlap. However, they might be effective in different ways (i.e., direction of rubbing &amp; viewing); and they might be picked up differently, depending on the sensitivity to different sorts of statistics, the resolution of the sensor, etc. So using both vision and touch will get one closer to relevant physical properties than each modality in isolation.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Difficulties arising with &#8220;veridical&#8221; perceptions<\/h3>\n\n\n\n<p style=\"text-align:left\">Should we consider that there is an absolute physical world with an extensive set of properties and affordances, ready to be picked up by a perceiving subject (human or animal)? Even if it were the case, there seems to be little use of such an absolute vision of the world (or God&#8217;s Eye according to Koenderink). Indeed, only considering the scale dimension, this would mean that all possible scales should be computed for all perceiving organisms (that have existed, exist or will exist), and for all possible visual tasks (seeing from up close, at arm&#8217;s length, at a distance, etc). This sounds like a daunting and unnecessary task&#8230;<\/p>\n\n\n\n<p>What then should be considered as &#8220;veridical&#8221;? The opposite approach would be to consider that nothing is of the sort. But then how to understand that with prolonged observation, one identifies details that he was  previously unaware of, and that this increased awareness can be communicated to others? What happens when one gets closer to or turn an object in his hand to reveal shape details that previously went unnoticed? Surely, these newly discovered features take the observer closer to a richer (more veridical?) appreciation of his world; not only his world, but also the world of members of his species. This is the key idea in the distinction between potential, effective and picked up cues. <\/p>\n\n\n\n<p>Can we go &#8220;more veridical&#8221; than what&#8217;s potentially available to prolonged scrutiny? One may use an optical apparatus to discover details at larger distances or smaller scales, hence uncovering fascinating new worlds! But then this is just expanding the set of potentially available properties of the world through vision; and while one may observe other scales in the process, this will be done at the expense of other aspects of the world that are temporarily set aside. If instead we venture away from vision, then one may measure properties of the world (shape, materials, distances, etc) in a viewpoint-independent manner. Isn&#8217;t this an appropriate veridical description of the world? No, because it still depends on a choice of scale: a surface only exists as an abstraction, a material depends on a scale above which variations of matter become shape details, etc. <\/p>\n\n\n\n<p>Alternatively, we might say that the world may be described veridically from a subjective viewpoint. Sounds like a paradox, hence we might be better off speaking of instantiated physical models (of shape, material, space, dynamics) at a current scale of observation, of the potential visual cues they are able to generate once a specific viewpoint is chosen, of the effectively available cues that arise in the ambient optic array once lighting and motion are specified, and of the subset of these cues that are actually perceived by a living organism. Actually, this should be presented the other way around: the organism, depending on a variety of factors, picks up some of the effectively available cues, with enough experience probes the environment to turn available potential cues into more effective cues appropriate to the current task, and with sufficient education learns to relate this subjective visual probing of the environment to measurable properties through means other than his own senses, which is scientific inquiry. <\/p>\n\n\n\n<p>I hope that by considering all these aspects, it might be possible to re-conciliate Gibson with von Uexkull: each organism leaves in his own world-bubble, which overlaps significantly with his peers of the same species, and to lesser extents with other animals; in this context, and at any given moment and scale, one might characterize what properties of the world-bubble are potentially available visually, then effectively found in the ambient optic array to be picked up by the organism; the process is dynamic in that the animal will likely probe his world-bubble to turn potential cues into effective ones, or even change the very nature of the world-bubble by getting closer or (for humans) using optical aids to increase visual awareness. The same approach should extent to other sensory modalities.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Several questions remain<\/h3>\n\n\n\n<p>Who should choose which potential\/effective cues to study? Should &#8220;scientific&#8221; intuition be the only source or are they more systematic ways to decide what matters? Maybe the decision should be based on actual behaviors that we want to understand, and for which we study the specific network of effective\/potential cues, taking special care to observe variations in behaviors that could be explained by different picking\/probing strategies?<\/p>\n\n\n\n<p>What is the role of development and learning of the organism in the picking up\/probing of information? This is particularly interesting since if we can understand how different strategies evolve through the life of a human individual, we may then try to transpose the method to the study of other animals&#8230;<\/p>\n\n\n\n<p>What is attributed to perception, cognition or reflective thoughts in a visual task? This is a question specific to humans once one assumes that language is necessary to reflective thoughts. What about cognition? <\/p>\n\n\n\n<p>How to generalize to other animals than humans? This is one of the hardest questions. If we assume that there is no possible veridical description of the world, this sounds like an impossible task. The best we can do seems to identify the right scales at which the animal operates, transpose our own awareness to this setup (maybe with optical aids for vision), and hypothetize which effective\/potential cues might be probed\/picked up for a given task. This will not tell us &#8220;what it is to be &#8230;&#8221; some animal, but it might help making analogies with our own awareness, which might well be the best we can hope for&#8230;<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In praise of Ecological Optics Stimuli are usually generated via interactions of light, shape and matter through rendering equation. But only gives information about 0th-order image structure (pixel intensity and color)! Some studies connect perception directly to physical properties, others to mid-level factors.&nbsp;This raises new questions about how these properties are detected in the observed &#8230; <a title=\"Mechanistic visual perception?\" class=\"read-more\" href=\"https:\/\/www.labri.fr\/perso\/barla\/blog\/?p=39754\" aria-label=\"Read more about Mechanistic visual perception?\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[651],"tags":[],"class_list":["post-39754","post","type-post","status-publish","format-standard","hentry","category-thoughts"],"_links":{"self":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/39754","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=39754"}],"version-history":[{"count":15,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/39754\/revisions"}],"predecessor-version":[{"id":40078,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/39754\/revisions\/40078"}],"wp:attachment":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=39754"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=39754"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=39754"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}