Ran Gal - Academia.edu (original) (raw)
Papers by Ran Gal
Emerging platforms such as Kinect, Epson Moverio, or Meta SpaceGlasses enable immersive experienc... more Emerging platforms such as Kinect, Epson Moverio, or Meta SpaceGlasses enable immersive experiences, where applications display content on multiple walls and multiple devices, detect objects in the world, and display content near those objects. App stores for these platforms enable users to run applications from third parties. Unfortunately, to display content properly near objects and on room surfaces, these applications need highly sensitive information, such as video and depth streams from the room, thus creating a serious privacy problem for app users. To solve this problem, we introduce two new abstractions enabling least privilege interactions of apps with the room. First, a room skeleton that provides least privilege for ren-dering, unlike previous approaches that focus on inputs alone. Second, a detection sandbox that allows registering content to show if an object is detected, but prevents the application from knowing if the object is present. To demonstrate our ideas, we h...
Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wir... more Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wires (in green) are extracted. Editing a few wires induces a new wire configuration (in blue) and leads to the result on the right. Man-made objects are largely dominated by a few typical features that carry special characteristics and engineered meanings. Stateof-the-art deformation tools fall short at preserving such characteristic features and global structure. We introduce iWIRES, a novel approach based on the argument that man-made models can be distilled using a few special 1D wires and their mutual relations. We hypothesize that maintaining the properties of such a small number of wires allows preserving the defining characteristics of the entire object. We introduce an analyze-and-edit approach, where prior to editing, we perform a light-weight analysis of the input shape to extract a descriptive set of wires. Analyzing the individual and mutual properties of the wires, and augment...
The ability of computer graphics to represent images symbolically has so far been used mostly to ... more The ability of computer graphics to represent images symbolically has so far been used mostly to render existing models with greater clarity or with greater visual appeal. In this work, we present a method aimed at harnessing this symbolic representation power
This article introduces a method for mesh editing that is aimed at preserving shape and volume. W... more This article introduces a method for mesh editing that is aimed at preserving shape and volume. We present two new developments: The first is a minimization of a functional expressing a geometric distance measure between two isometric surfaces. The second is a local volume analysis linking the volume of an object to its surface curvature. Our method is based upon the moving frames representation of meshes. Applying a rotation field to the moving frames defines an isometry. Given rotational constraints, the mesh is deformed by an optimal isometry defined by minimizing the distance measure between original and deformed meshes. The resulting isometry nicely preserves the surface details, but when large rotations are applied, the volumetric behavior of the model may be unsatisfactory. Using the local volume analysis, we define a scalar field by which we scale the moving frames. Scaled and rotated moving frames restore volumetric properties of the original mesh, while properly maintainin...
Abstract—A 3D shape signature is a compact representation for some essence of a shape. Shape sign... more Abstract—A 3D shape signature is a compact representation for some essence of a shape. Shape signatures are commonly utilized as a fast indexing mechanism for shape retrieval. Effective shape signatures capture some global geometric properties which are scale, translation, and rotation invariant. In this paper, we introduce an effective shape signature which is also pose-oblivious. This means that the signature is also insensitive to transformations which change the pose of a 3D shape such as skeletal articulations. Although some topology-based matching methods can be considered pose-oblivious as well, our new signature retains the simplicity and speed of signature indexing. Moreover, contrary to topology-based methods, the new signature is also insensitive to the topology change of the shape, allowing us to match similar shapes with different genus. Our shape signature is a 2D histogram which is a combination of the distribution of two scalar functions defined on the boundary surfa...
This article introduces a method for partial matching of surfaces represented by triangular meshe... more This article introduces a method for partial matching of surfaces represented by triangular meshes. Our method matches surface regions that are numerically and topologically dissimilar, but approximately similar regions. We introduce novel local surface descriptors which efficiently represent the geometry of local regions of the surface. The descriptors are defined independently of the underlying triangulation, and form a compatible representation that allows matching of surfaces with different triangulations. To cope with the combinatorial complexity of partial matching of large meshes, we introduce the abstraction of salient geometric features and present a method to construct them. A salient geometric feature is a compound high-level feature of nontrivial local shapes. We show that a relatively small number of such salient geometric features characterizes the surface well for various similarity applications. Matching salient geometric features is based on indexing rotation-invari...
The ability of computer graphics to represent images symbolically has so far been used mostly to ... more The ability of computer graphics to represent images symbolically has so far been used mostly to render existing models with greater clarity or with greater visual appeal. In this work, we present a method aimed at harnessing this symbolic representation power to increase the expressiveness of the 3D models themselves. We achieve this through modification of the actual representation of 3D shapes rather than their images. In particular, we focus on 3D collage creation, namely, a generation of compound representations of objects. The ability of such representations to convey multiple meanings has been recognized for centuries. At the same time, it has also been acknowledged that for humans, the creation of compound 3D shapes is extremely taxing. Thus, this expressive but technically challenging artistic medium is a particularly good candidate to address using computer graphics methods. We present an algorithm for 3D collage generation that serves as an artistic tool performing the ch...
We present a method for inhomogeneous 2D texture mapping guided by a feature mask, that preserves... more We present a method for inhomogeneous 2D texture mapping guided by a feature mask, that preserves some regions of the image, such as foreground objects or other prominent parts. The method is able to arbitrarily warp a given image while preserving the shape of its features by constraining their deformation to be a similarity transformation. In particular, our method allows global or local changes to the aspect ratio of the texture without causing undesirable shearing to the features. The algorithmic core of our method is a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain. The method is useful in digital imaging, texture design and any other applications involving image warping, where parts of the image have high familiarity and should retain their shape after modification.
We present an example-based surface reconstruction method for scanned point sets. Our approach us... more We present an example-based surface reconstruction method for scanned point sets. Our approach uses a database of local shape priors built from a set of given context models that are chosen specifically to match a specific scan. Local neighborhoods of the input scan are matched with enriched patches of these models at multiple scales. Hence, instead of using a single prior for reconstruction, our method allows specific regions in the scan to match the most relevant prior that fits best. Such high confidence matches carry relevant information from the prior models to the scan, including normal data and feature classification, and are used to augment the input point-set. This allows to resolve many ambiguities and difficulties that come up during reconstruction, e.g., distinguishing between signal and noise or between gaps in the data and boundaries of the model. We demonstrate how our algorithm, given suitable prior models, successfully handles noisy and under-sampled point sets, fai...
Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wir... more Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wires (in green) are extracted. Editing a few wires induces a new wire configuration (in blue) and leads to the result on the right. Man-made objects are largely dominated by a few typical features that carry special characteristics and engineered meanings. Stateof-the-art deformation tools fall short at preserving such characteristic features and global structure. We introduce iWIRES, a novel approach based on the argument that man-made models can be distilled using a few special 1D wires and their mutual relations. We hypothesize that maintaining the properties of such a small number of wires allows preserving the defining characteristics of the entire object. We introduce an analyze-and-edit approach, where prior to editing, we perform a light-weight analysis of the input shape to extract a descriptive set of wires. Analyzing the individual and mutual properties of the wires, and augment...
Abstract-Immersive experiences that mix digital and real-world objects are becoming reality, but ... more Abstract-Immersive experiences that mix digital and real-world objects are becoming reality, but they raise serious privacy concerns as they require real-time sensor input. These experiences are already present on smartphones and game consoles via Kinect, and will eventually emerge on the web platform. However, browsers do not expose the display interfaces needed to render immersive experiences. Previous security research focuses on controlling application access to sensor input alone, and do not deal with display interfaces. Recent research in human computer interactions has explored a variety of high-level rendering interfaces for immersive experiences, but these interfaces reveal sensitive data to the application. Bringing immersive experiences to the web requires a high-level interface that mitigates privacy concerns. This paper presents SurroundWeb, the first 3D web browser, which provides the novel functionality of rendering web content onto a room while tackling many of the i...
The different simulator parameters optimized for the Clevr experiment and the underlying distribu... more The different simulator parameters optimized for the Clevr experiment and the underlying distributions are shown in Table 1. The validation set of 30 images is composed of synthetic images generated with a particular simulator configuration which involved a particular lighting configuration, quality of images, image size, location and material of images. This is shown in the right column of the last row of Fig.1. This configuration is hidden during the simulator training and only the validation set is available.
1 We present a new approach for rule-based design layout prob2 lems, examples of which include au... more 1 We present a new approach for rule-based design layout prob2 lems, examples of which include automated graphic design, fur3 niture arrangement and synthesizing virtual worlds. We formu4 late this as a weighted constraint-satisfaction problem which is 5 high-dimensional and requires non-convex optimization. We use 6 a hyper-graph to represent a set of objects and the design rules ap7 plied to them. Although inference in graphs with higher order edges 8 is computationally expensive, we present a hybrid solution-space 9 exploration algorithm that adaptively represents higher order rela10 tions compactly. We exploit the fact that for many types of rules, 11 satisfiable assignments can be found efficiently. In other words, 12 these rules are locally satisfiable. Therefore we can sample from 13 the known partial probability distribution function for each rule. 14 Experimental results demonstrate that this sampling technique re15 duces the number of samples required by other algorithms b...
Measuring Visual Latency in VR and AR devices has become increasingly complicated as many of the ... more Measuring Visual Latency in VR and AR devices has become increasingly complicated as many of the components will influence others in multiple loops and ultimately affect the human cognitive and sensory perception. In this paper we present a new method based on the idea that the performance of humans on a rapid motor task will remain constant, and that any added delay will correspond to the system latency. We ask users to perform a task inside different video see-through devices and also in front of a computer. We also calculate the latency of the systems using a hardware instrumentation-based measurement technique for bench-marking. Results show that this new form of latency measurement through human cognitive performance can be reliable and comparable to hardware instrumentation-based measurement. Our method is adaptable to many forms of user interaction. It is particularly suitable for systems, such as AR and VR, where externalizing signals is difficult, or where it is important t...
1 Creating a layout for an augmented reality (AR) application which 2 embeds virtual objects in a... more 1 Creating a layout for an augmented reality (AR) application which 2 embeds virtual objects in a physical environment is difficult as it 3 must adapt to any physical space. We propose a rule-based frame4 work for generating object layouts for AR applications. We present 5 an algorithm for dynamically targeting the AR application to a new 6 environment in real time by solving a constraint-satisfaction problem. 7 Under our framework, the developer of an AR application specifies 8 a set of cost functions (rules) which enforce self-consistency (rules 9 regarding the inter-relationships of application components) and 10 scene-consistency (application components are consistent with the 11 physical environment they are placed in). Our method is general 12 and can be applied to any rule-based layout design problems. We 13 represent layout rules using hyper-graphs where nodes in the graph 14 represent objects and hyper-edges between nodes represent the rules 15 that operate on objects. 16 G...
2019 IEEE International Conference on Image Processing (ICIP), Sep 1, 2019
We present an approach to synthesize highly photorealistic images of 3D object models, which we u... more We present an approach to synthesize highly photorealistic images of 3D object models, which we use to train a convolutional neural network for detecting the objects in real images. The proposed approach has three key ingredients: (1) 3D object models are rendered in 3D models of complete scenes with realistic materials and lighting, (2) plausible geometric configuration of objects and cameras in a scene is generated using physics simulation, and (3) high photorealism of the synthesized images is achieved by physically based rendering. When trained on images synthesized by the proposed approach, the Faster R-CNN object detector [1] achieves a 24% absolute improvement of mAP@.75IoU on Rutgers APC [2] and 11% on LineMod-Occluded [3] datasets, compared to a baseline where the training images are synthesized by rendering object models on top of random photographs. This work is a step towards being able to effectively train object detectors without capturing or annotating any real images. A dataset of 600K synthetic images with ground truth annotations for various computer vision tasks will be released on the project website: thodan.github.io/objectsynth.
Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game... more Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game is difficult. In our framework (a) declarative rules are used to define application elements and the rules governing them (b) in real-time we analyze an environment to extract scene geometry and horizontal and vertical planes (c) our move-making algorithm targets the application to the room (d) an additional result of our system in a different room with a longer track. ABSTRACT Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based framework for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regarding the interrelationships of application components) and scene-consistency (application components are consistent with the physical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possible). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making algorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.
Lecture Notes in Computer Science, 2014
2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2014
ABSTRACT Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic ra... more ABSTRACT Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game is difficult. In our framework (a) declarative rules are used to define application elements and the rules governing them (b) in real-time we analyze an environment to extract scene geometry and horizontal and vertical planes (c) our move-making algorithm targets the application to the room (d) an additional result of our system in a different room with a longer track. ABSTRACT Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based frame-work for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regard-ing the inter-relationships of application components) and scene-consistency (application components are consistent with the phys-ical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possi-ble). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making al-gorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.
Emerging platforms such as Kinect, Epson Moverio, or Meta SpaceGlasses enable immersive experienc... more Emerging platforms such as Kinect, Epson Moverio, or Meta SpaceGlasses enable immersive experiences, where applications display content on multiple walls and multiple devices, detect objects in the world, and display content near those objects. App stores for these platforms enable users to run applications from third parties. Unfortunately, to display content properly near objects and on room surfaces, these applications need highly sensitive information, such as video and depth streams from the room, thus creating a serious privacy problem for app users. To solve this problem, we introduce two new abstractions enabling least privilege interactions of apps with the room. First, a room skeleton that provides least privilege for ren-dering, unlike previous approaches that focus on inputs alone. Second, a detection sandbox that allows registering content to show if an object is detected, but prevents the application from knowing if the object is present. To demonstrate our ideas, we h...
Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wir... more Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wires (in green) are extracted. Editing a few wires induces a new wire configuration (in blue) and leads to the result on the right. Man-made objects are largely dominated by a few typical features that carry special characteristics and engineered meanings. Stateof-the-art deformation tools fall short at preserving such characteristic features and global structure. We introduce iWIRES, a novel approach based on the argument that man-made models can be distilled using a few special 1D wires and their mutual relations. We hypothesize that maintaining the properties of such a small number of wires allows preserving the defining characteristics of the entire object. We introduce an analyze-and-edit approach, where prior to editing, we perform a light-weight analysis of the input shape to extract a descriptive set of wires. Analyzing the individual and mutual properties of the wires, and augment...
The ability of computer graphics to represent images symbolically has so far been used mostly to ... more The ability of computer graphics to represent images symbolically has so far been used mostly to render existing models with greater clarity or with greater visual appeal. In this work, we present a method aimed at harnessing this symbolic representation power
This article introduces a method for mesh editing that is aimed at preserving shape and volume. W... more This article introduces a method for mesh editing that is aimed at preserving shape and volume. We present two new developments: The first is a minimization of a functional expressing a geometric distance measure between two isometric surfaces. The second is a local volume analysis linking the volume of an object to its surface curvature. Our method is based upon the moving frames representation of meshes. Applying a rotation field to the moving frames defines an isometry. Given rotational constraints, the mesh is deformed by an optimal isometry defined by minimizing the distance measure between original and deformed meshes. The resulting isometry nicely preserves the surface details, but when large rotations are applied, the volumetric behavior of the model may be unsatisfactory. Using the local volume analysis, we define a scalar field by which we scale the moving frames. Scaled and rotated moving frames restore volumetric properties of the original mesh, while properly maintainin...
Abstract—A 3D shape signature is a compact representation for some essence of a shape. Shape sign... more Abstract—A 3D shape signature is a compact representation for some essence of a shape. Shape signatures are commonly utilized as a fast indexing mechanism for shape retrieval. Effective shape signatures capture some global geometric properties which are scale, translation, and rotation invariant. In this paper, we introduce an effective shape signature which is also pose-oblivious. This means that the signature is also insensitive to transformations which change the pose of a 3D shape such as skeletal articulations. Although some topology-based matching methods can be considered pose-oblivious as well, our new signature retains the simplicity and speed of signature indexing. Moreover, contrary to topology-based methods, the new signature is also insensitive to the topology change of the shape, allowing us to match similar shapes with different genus. Our shape signature is a 2D histogram which is a combination of the distribution of two scalar functions defined on the boundary surfa...
This article introduces a method for partial matching of surfaces represented by triangular meshe... more This article introduces a method for partial matching of surfaces represented by triangular meshes. Our method matches surface regions that are numerically and topologically dissimilar, but approximately similar regions. We introduce novel local surface descriptors which efficiently represent the geometry of local regions of the surface. The descriptors are defined independently of the underlying triangulation, and form a compatible representation that allows matching of surfaces with different triangulations. To cope with the combinatorial complexity of partial matching of large meshes, we introduce the abstraction of salient geometric features and present a method to construct them. A salient geometric feature is a compound high-level feature of nontrivial local shapes. We show that a relatively small number of such salient geometric features characterizes the surface well for various similarity applications. Matching salient geometric features is based on indexing rotation-invari...
The ability of computer graphics to represent images symbolically has so far been used mostly to ... more The ability of computer graphics to represent images symbolically has so far been used mostly to render existing models with greater clarity or with greater visual appeal. In this work, we present a method aimed at harnessing this symbolic representation power to increase the expressiveness of the 3D models themselves. We achieve this through modification of the actual representation of 3D shapes rather than their images. In particular, we focus on 3D collage creation, namely, a generation of compound representations of objects. The ability of such representations to convey multiple meanings has been recognized for centuries. At the same time, it has also been acknowledged that for humans, the creation of compound 3D shapes is extremely taxing. Thus, this expressive but technically challenging artistic medium is a particularly good candidate to address using computer graphics methods. We present an algorithm for 3D collage generation that serves as an artistic tool performing the ch...
We present a method for inhomogeneous 2D texture mapping guided by a feature mask, that preserves... more We present a method for inhomogeneous 2D texture mapping guided by a feature mask, that preserves some regions of the image, such as foreground objects or other prominent parts. The method is able to arbitrarily warp a given image while preserving the shape of its features by constraining their deformation to be a similarity transformation. In particular, our method allows global or local changes to the aspect ratio of the texture without causing undesirable shearing to the features. The algorithmic core of our method is a particular formulation of the Laplacian editing technique, suited to accommodate similarity constraints on parts of the domain. The method is useful in digital imaging, texture design and any other applications involving image warping, where parts of the image have high familiarity and should retain their shape after modification.
We present an example-based surface reconstruction method for scanned point sets. Our approach us... more We present an example-based surface reconstruction method for scanned point sets. Our approach uses a database of local shape priors built from a set of given context models that are chosen specifically to match a specific scan. Local neighborhoods of the input scan are matched with enriched patches of these models at multiple scales. Hence, instead of using a single prior for reconstruction, our method allows specific regions in the scan to match the most relevant prior that fits best. Such high confidence matches carry relevant information from the prior models to the scan, including normal data and feature classification, and are used to augment the input point-set. This allows to resolve many ambiguities and difficulties that come up during reconstruction, e.g., distinguishing between signal and noise or between gaps in the data and boundaries of the model. We demonstrate how our algorithm, given suitable prior models, successfully handles noisy and under-sampled point sets, fai...
Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wir... more Figure 1: A complex model (left) consisting of 108 components is analyzed and 250 intelligent wires (in green) are extracted. Editing a few wires induces a new wire configuration (in blue) and leads to the result on the right. Man-made objects are largely dominated by a few typical features that carry special characteristics and engineered meanings. Stateof-the-art deformation tools fall short at preserving such characteristic features and global structure. We introduce iWIRES, a novel approach based on the argument that man-made models can be distilled using a few special 1D wires and their mutual relations. We hypothesize that maintaining the properties of such a small number of wires allows preserving the defining characteristics of the entire object. We introduce an analyze-and-edit approach, where prior to editing, we perform a light-weight analysis of the input shape to extract a descriptive set of wires. Analyzing the individual and mutual properties of the wires, and augment...
Abstract-Immersive experiences that mix digital and real-world objects are becoming reality, but ... more Abstract-Immersive experiences that mix digital and real-world objects are becoming reality, but they raise serious privacy concerns as they require real-time sensor input. These experiences are already present on smartphones and game consoles via Kinect, and will eventually emerge on the web platform. However, browsers do not expose the display interfaces needed to render immersive experiences. Previous security research focuses on controlling application access to sensor input alone, and do not deal with display interfaces. Recent research in human computer interactions has explored a variety of high-level rendering interfaces for immersive experiences, but these interfaces reveal sensitive data to the application. Bringing immersive experiences to the web requires a high-level interface that mitigates privacy concerns. This paper presents SurroundWeb, the first 3D web browser, which provides the novel functionality of rendering web content onto a room while tackling many of the i...
The different simulator parameters optimized for the Clevr experiment and the underlying distribu... more The different simulator parameters optimized for the Clevr experiment and the underlying distributions are shown in Table 1. The validation set of 30 images is composed of synthetic images generated with a particular simulator configuration which involved a particular lighting configuration, quality of images, image size, location and material of images. This is shown in the right column of the last row of Fig.1. This configuration is hidden during the simulator training and only the validation set is available.
1 We present a new approach for rule-based design layout prob2 lems, examples of which include au... more 1 We present a new approach for rule-based design layout prob2 lems, examples of which include automated graphic design, fur3 niture arrangement and synthesizing virtual worlds. We formu4 late this as a weighted constraint-satisfaction problem which is 5 high-dimensional and requires non-convex optimization. We use 6 a hyper-graph to represent a set of objects and the design rules ap7 plied to them. Although inference in graphs with higher order edges 8 is computationally expensive, we present a hybrid solution-space 9 exploration algorithm that adaptively represents higher order rela10 tions compactly. We exploit the fact that for many types of rules, 11 satisfiable assignments can be found efficiently. In other words, 12 these rules are locally satisfiable. Therefore we can sample from 13 the known partial probability distribution function for each rule. 14 Experimental results demonstrate that this sampling technique re15 duces the number of samples required by other algorithms b...
Measuring Visual Latency in VR and AR devices has become increasingly complicated as many of the ... more Measuring Visual Latency in VR and AR devices has become increasingly complicated as many of the components will influence others in multiple loops and ultimately affect the human cognitive and sensory perception. In this paper we present a new method based on the idea that the performance of humans on a rapid motor task will remain constant, and that any added delay will correspond to the system latency. We ask users to perform a task inside different video see-through devices and also in front of a computer. We also calculate the latency of the systems using a hardware instrumentation-based measurement technique for bench-marking. Results show that this new form of latency measurement through human cognitive performance can be reliable and comparable to hardware instrumentation-based measurement. Our method is adaptable to many forms of user interaction. It is particularly suitable for systems, such as AR and VR, where externalizing signals is difficult, or where it is important t...
1 Creating a layout for an augmented reality (AR) application which 2 embeds virtual objects in a... more 1 Creating a layout for an augmented reality (AR) application which 2 embeds virtual objects in a physical environment is difficult as it 3 must adapt to any physical space. We propose a rule-based frame4 work for generating object layouts for AR applications. We present 5 an algorithm for dynamically targeting the AR application to a new 6 environment in real time by solving a constraint-satisfaction problem. 7 Under our framework, the developer of an AR application specifies 8 a set of cost functions (rules) which enforce self-consistency (rules 9 regarding the inter-relationships of application components) and 10 scene-consistency (application components are consistent with the 11 physical environment they are placed in). Our method is general 12 and can be applied to any rule-based layout design problems. We 13 represent layout rules using hyper-graphs where nodes in the graph 14 represent objects and hyper-edges between nodes represent the rules 15 that operate on objects. 16 G...
2019 IEEE International Conference on Image Processing (ICIP), Sep 1, 2019
We present an approach to synthesize highly photorealistic images of 3D object models, which we u... more We present an approach to synthesize highly photorealistic images of 3D object models, which we use to train a convolutional neural network for detecting the objects in real images. The proposed approach has three key ingredients: (1) 3D object models are rendered in 3D models of complete scenes with realistic materials and lighting, (2) plausible geometric configuration of objects and cameras in a scene is generated using physics simulation, and (3) high photorealism of the synthesized images is achieved by physically based rendering. When trained on images synthesized by the proposed approach, the Faster R-CNN object detector [1] achieves a 24% absolute improvement of mAP@.75IoU on Rutgers APC [2] and 11% on LineMod-Occluded [3] datasets, compared to a baseline where the training images are synthesized by rendering object models on top of random photographs. This work is a step towards being able to effectively train object detectors without capturing or annotating any real images. A dataset of 600K synthetic images with ground truth annotations for various computer vision tasks will be released on the project website: thodan.github.io/objectsynth.
Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game... more Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game is difficult. In our framework (a) declarative rules are used to define application elements and the rules governing them (b) in real-time we analyze an environment to extract scene geometry and horizontal and vertical planes (c) our move-making algorithm targets the application to the room (d) an additional result of our system in a different room with a longer track. ABSTRACT Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based framework for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regarding the interrelationships of application components) and scene-consistency (application components are consistent with the physical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possible). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making algorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.
Lecture Notes in Computer Science, 2014
2014 IEEE International Symposium on Mixed and Augmented Reality (ISMAR), 2014
ABSTRACT Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic ra... more ABSTRACT Figure 1: Designing an immersive augmented reality (AR) application such as a dynamic racing game is difficult. In our framework (a) declarative rules are used to define application elements and the rules governing them (b) in real-time we analyze an environment to extract scene geometry and horizontal and vertical planes (c) our move-making algorithm targets the application to the room (d) an additional result of our system in a different room with a longer track. ABSTRACT Creating a layout for an augmented reality (AR) application which embeds virtual objects in a physical environment is difficult as it must adapt to any physical space. We propose a rule-based frame-work for generating object layouts for AR applications. Under our framework, the developer of an AR application specifies a set of rules (constraints) which enforce self-consistency (rules regard-ing the inter-relationships of application components) and scene-consistency (application components are consistent with the phys-ical environment they are placed in). When a user enters a new environment, we create, in real-time, a layout for the application, which is consistent with the defined constraints (as much as possi-ble). We find the optimal configurations for each object by solving a constraint-satisfaction problem. Our stochastic move making al-gorithm is domain-aware, and allows us to efficiently converge to a solution for most rule-sets. In the paper we demonstrate several augmented reality applications that automatically adapt to different rooms and changing circumstances in each room.