{"id":40263,"date":"2020-01-06T09:48:57","date_gmt":"2020-01-06T09:48:57","guid":{"rendered":"http:\/\/www.labri.fr\/perso\/barla\/blog\/?p=40263"},"modified":"2020-01-06T11:29:59","modified_gmt":"2020-01-06T11:29:59","slug":"proximal-generative-framework","status":"publish","type":"post","link":"https:\/\/www.labri.fr\/perso\/barla\/blog\/?p=40263","title":{"rendered":"Proximal generative framework"},"content":{"rendered":"<p id=\"top\" \/>\n<p>Trying to explain how the (or rather a) physical world is perceived by a human or animal subject seems like a huge leap to perform. Intermediate steps are required to understand which of the intricacies of matter on various scales come to be perceived by an inherently limited brain, and how.<\/p>\n\n\n\n<p>The idea of the proximal generative framework is to bring all the relevant visual information provided by the environment up to the sensor, just before visual perception can take place. It is a bottom-up process of progressive simplification and organization, starting from the <em>structuration of the environment at various spatial and temporal scales<\/em> studied via physics, and going through the <em>structuration of images for various viewing and lighting configurations<\/em> which constitutes the under-explored field of ecological optics. Vision may then work from this pictorial information that reaches the eye, most likely in a dynamic and top-down fashion; it will  consist in a <em>structure-driven perception in the context of a specific human or animal activity.<\/em> <\/p>\n\n\n\n<p>Whereas visual perception can vary tremendously depending on the observer, his activity and state of mind, and is thus inherently subjective, the visual structuration of the environment can be considered an objective process provided the physical world we start from is properly defined. This requires to pick an inner scale that separates what is not visible to the naked eye has to be dealt with by physics (the back end), while all structuration above that scale will be studied through ecological optics (the front end). The choice of scale itself is subjective, and may vary with the distance of objects to the observer or even with attention; this is an inescapable chicken and egg problem that can only be escaped by positing that the graininess of the physical world is made to change dynamically for the sake of our analysis (i.e., it&#8217;s a mental construct, like any science is).<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The back end<\/h3>\n\n\n\n<p>Various scales of interest, from quantum scales for emission and reflectance spectra, to nano-scale for photonic crystals, micro-scale for statistical roughness properties, up to even bigger scales if the choice of inner scale is big enough (think of a mountain).<\/p>\n\n\n\n<p>Not only electro-magnetism in the visible wavelength range, but also mechanical properties of objects that may manifest themselves visually.   Include growth patterns, material layers, iridescence, scattering, surface tension, reflectance, transmittance, absorbance, elasticity, plasticity, translucency, viscosity, etc.<\/p>\n\n\n\n<p>Main role of the back end is to convert these physical properties into visual cues, simplifying their complexity to match the resolution of the inner scale.  But do not consider actual viewpoint or lighting yet: only study potential cues that <em>could occur locally<\/em> (in space and time), at the chosen inner scale. One example is BSDF: considers all potential lighting and viewing directions for scattering &#8220;at&#8221; a point. It&#8217;s thus only a 0-order description; descriptions over infinitesimal neighborhoods, or even extended neighborhoods should be sought for. Multiple layers should be identified (e.g., specular over diffuse) and kept separated. Local potential motion should also be studied, depending on optical and mechanical properties (think of a shiny viscous fluid for instance).<\/p>\n\n\n\n<p>Outcome of this stage is a summary of physical properties at the chosen inner scale in terms of potential visual cues lying on surfaces in 3D.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">The front end<\/h3>\n\n\n\n<p>What happens when a viewpoint is specified and objects are placed in an environment that acts as a whole source of illumination? Usually the topic of rendering in computer graphics, when it comes to generate a synthetic color image. But here we&#8217;re interested in how the structure that came from the back end &#8212; the potential cues &#8212; is filtered, modified even augmented by the structuration due lighting and projection toward the viewpoint.<\/p>\n\n\n\n<p>In the language of ecological optics, describe structure of the ambient optic array starting from what we know of the spatio-temporal structure <em>at<\/em> the inner scale. Structures include: reflection distorsions, filters, colors, motion transparency, motion perspective, congruence, correlations toward contours, optic flow properties, inertial patterns, etc.<\/p>\n\n\n\n<p>Main role of the front end is to turn potentially available cues defined on surfaces into effectively available cues defined on the ambient optic array. For instance, the lighting environment will likely affect how BSDF components are exaggerated or masked (0-th order scale), but also how highlights get distorted by surface shape (defined at the inner scale), and even how they conform to the occluding contours produced by perspective projection. Potential motions may similarly be made effective by a motion of the observer or the object itself, and integrated over extended times to reveal layout cues and mechanical cues.<\/p>\n\n\n\n<p>The outcome of this stage is a new summary, this time in terms of effective cues in the image plane (or ambien optic array), that are not only subset of potential cues, but also connected over extended spatial and temporal behaviors and reveal scene layout through the emergence of contours.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Shortcomings<\/h3>\n\n\n\n<p>Not all physical worlds convey a structured pirctorial organization of the ambient optic array. In some cases, image patterns must be accepted as entirely or partly <em>unstructured<\/em>. Examples that come to mind are high densities of overlapping disparate objects, refraction patterns, multiple reflections, caustics and shadows. They can be made sense of through reflective thought or even scientific enquiry, but may as well be simply left unexplained. The visual system tends to do that: accept that the pattern is simply complex. In these cases, artists learn to simply record the patterns, and some take the liberty to simplify them, knowing it won&#8217;t impact much perception.<\/p>\n\n\n\n<p>This is to contrast with <em>astructured<\/em> regions, which are simply regions of the optic array that are devoid of variations: they are simply filled with a uniform color. The most common example is dark regions (the noistrills effect) that most usually do not let any structure appear; however, the pattern of progressive darkening in cavities is itself visually informative of shape and space. Other examples include a blue or overcast sky, fog, very bright patches, etc. They may manifest themselves as actual proximal color patches not intrinsically related to the scene (think of glow effects).<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Trying to explain how the (or rather a) physical world is perceived by a human or animal subject seems like a huge leap to perform. Intermediate steps are required to understand which of the intricacies of matter on various scales come to be perceived by an inherently limited brain, and how. The idea of the &#8230; <a title=\"Proximal generative framework\" class=\"read-more\" href=\"https:\/\/www.labri.fr\/perso\/barla\/blog\/?p=40263\" aria-label=\"Read more about Proximal generative framework\">Read more<\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[651],"tags":[],"class_list":["post-40263","post","type-post","status-publish","format-standard","hentry","category-thoughts"],"_links":{"self":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/40263","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=40263"}],"version-history":[{"count":6,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/40263\/revisions"}],"predecessor-version":[{"id":40270,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=\/wp\/v2\/posts\/40263\/revisions\/40270"}],"wp:attachment":[{"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=40263"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=40263"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.labri.fr\/perso\/barla\/blog\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=40263"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}