Green Screen Augmentation Enables Scene Generalisation in Robotic Manipulation

Abstract

Generalising vision-based manipulation policies to novel environments remains a challenging area with limited exploration. Current practices involve collecting data in one location, training imitation learning or reinforcement learning policies with this data, and deploying the policy in the same location. However, this approach lacks scalability as it necessitates data collection in multiple locations for each task. This paper proposes a novel approach where data is collected in a location predominantly featuring green screens. We introduce Green-screen Augmentation (GreenAug), employing a chroma key algorithm to overlay background textures onto a green screen. Through extensive real-world empirical studies with over 850 training demonstrations and 8.2k evaluation episodes, we demonstrate that GreenAug surpasses no augmentation, standard computer vision augmentation, and prior generative augmentation methods in performance. While no algorithmic novelties are claimed, our paper advocates for a fundamental shift in data collection practices. We propose that real-world demonstrations in future research should utilise green screens, followed by the application of GreenAug. We believe GreenAug unlocks policy generalisation to visually distinct novel locations, addressing the current scene generalisation limitations in robot learning.

Keywords: Green Screen, Data Augmentation, Learning from Demonstration

Figure 1: GreenAug provides a simple visual augmentation to robot policies by first collecting data with a green screen, then augmenting it with different textures. The resulting policy can be transferred to unseen visually distinct novel locations (scenes).

1 Introduction

Recent advancements in robot learning policies [1, 2, 3, 4, 5, 6, 7] have shown significant capabilities in performing complex manipulation tasks. However, generalising these policies to new locations remains a substantial challenge due to the lack of diverse training datasets. Ideally, these datasets should include a wide variety of environments, such as diverse areas of homes. However, gathering real-world data from different scenes is difficult and costly. These scenes refer to visually distinct physical locations, such as an oven situated in different kitchens or a toilet placed in various homes. The difficulty of collecting diverse data necessitates more efficient use of existing datasets.

Generative augmentation approaches [8, 9, 10] have attempted to address this by using generative models [11, 12, 13] to augment robot datasets. However, these methods often require extensive manual tuning and face several challenges. This includes text prompt engineering, chaining multiple object detectors, segmenters and generative models, and problems with performance and processing speed. Additionally, they can be inaccurate in robotic settings—particularly in segmentation and inpainting from wrist camera views, potentially introducing noise into robot policies.

In light of these complications, we opt for a simpler yet effective alternative: green screens. The film industry has utilised green screens extensively [14, 15, 16, 17, 18], enabling the addition of virtual backgrounds to live footage. Inspired by these applications, we apply green screen technology to robotics, allowing robots to perform tasks in unfamiliar scenes not part of the training demonstration data.

Figure 2: The GreenAug process begins with acquiring a green screen mask using chroma keying. GreenAug-Rand applies random textures, GreenAug-Gen uses generative models to inpaint the background, and GreenAug-Mask learns a masking network to filter out the background.

In this paper, we introduce Green-screen Augmentation (GreenAug), a simple real-world visual augmentation method that uses green screen and chroma keying to replace backgrounds, applicable to RGB-based robot learning methods. We explore several variants of GreenAug, including the use of random textures (Fig. 1), backgrounds generated by generative models, and a background masking network to obscure the background during inference. By replacing backgrounds with various textures, it allows robot learning policies to be robust against changes in visual scenes and focus on crucial features in the image space.

We conducted extensive real-world experiments across eight challenging robotic manipulation tasks and six further studies, amounting to over 850 training demonstrations and 8.2k evaluation episodes. We evaluated the performance of control policies in unseen scenes for head-to-head comparisons on scene generalisation. We compared several variants of GreenAug against approaches with no augmentation, standard computer vision augmentations, and a generative augmentation [8, 9, 10] method. Our results show that GreenAug outperforms no augmentation by 65%, standard computer vision augmentation by 29% and generative augmentation by 21%.

2 Related Work

Visual augmentation in robotics. Visual augmentation is important in robotics for adapting to changing environments. Standard computer vision augmentations like random photometric distortion, cropping, shifting, convolutions and overlays have enhanced performance in imitation learning [19, 20] and reinforcement learning [1, 21, 2, 22, 23, 24]. However, most of these methods only apply simple photometric perturbations. Domain randomisation [25, 26, 27, 28, 29, 30, 31] enhances this by generating synthetic data with varied visual and physical dynamics parameters for simulation-to-reality (Sim2Real) transfer. Alternatively, methods like CACTI [8], GenAug [9], and ROSIE [10] use generative models such as Stable Diffusion [12] to diversify visual data directly on real-world data, bypassing the need for simulation.

Green screen in machine learning and robotics. Green screen has been traditionally used for film and video production [14, 15, 16, 17, 18]. In recent years, its application in machine learning has increased. Smirnov et al. [18] applied machine learning to improve the quality of chroma keying. Xu et al. [32], Sengupta et al. [33], Lin et al. [34, 35] explored machine learning techniques to replace green screens, enabling natural image matting without them. Schülein et al. [36] used green screens and chroma keying to replace backgrounds with clinical scenes to create synthetic data for medical clothing detection. In robotics, the use of green screens remains limited. Coates and Ng [37] employed it to develop a multi-camera object detector with synthetic data from chroma-keyed backgrounds.

3 Green Screen Augmentation

In this section, we provide a detailed introduction to GreenAug. The practical steps for GreenAug are as follows: (1) Green Screen Scene Setup; (2) GreenAug via Chroma Keying; (3) Training Robot Learning Policies. In the following sub-sections, we expand on each of these stages.

3.1 Green Screen Scene Setup

Figure 3: Physical steps for green-screen setup. Scene items can either be moved into the green screen, or the green screen can be brought to the scene.

The act of scene setup consists of obscuring the background (i.e. non-task relevant objects) with a green screen. There are several ways of achieving this, two of which are highlighted in Fig. 3 and described below. Once the scene has been set up, demonstration collection can begin.

Scene to Green Screen, where a permanent green screen area or room is established, and items can be moved into the green screen for data collection. This is the most common use case and includes tasks such as general pick-and-place, opening drawers, sweeping, pushing, etc.

Green Screen to Scene, where the green screen is brought to a fixed, unmovable object. Scenes that usually fall into this category are ones that require manipulating integrated or heavy objects, such as stacking dishwashers, opening ovens, and opening doors.

3.2 GreenAug via Chroma Keying

Chroma keying is a visual effects technique for layering two images or video streams together based on colour hues (chroma range). This technique is commonly used in video production and post-production to composite two frames or images together by removing a background colour (usually green or blue) from the foreground content, making it transparent. This allows for the insertion of a new background or visual element in place of the green or blue background. Many chroma key algorithms exist, but we opt for a simple algorithm proposed by Cannon [38]. Given the generated mask, several options are available for applying GreenAug. We provide three variants of GreenAug: Random (GreenAug-Rand), Generative (GreenAug-Gen) and Mask (GreenAug-Mask), illustrated in Fig. 2 and described in detail below.

GreenAug-Rand This variant applies a fixed set of random textures to the chroma-keyed background. Following research in domain randomisation [25, 26, 27, 28, 29, 30, 31], increasing the variability of these textures helps the policy ignore the background and focus on task-specific items (objects manipulated by the policy).

GreenAug-Gen. This variant uses the chroma-keyed mask to inpaint realistic or imagined backgrounds using generative models like Stable Diffusion. Examples of prompts include: “photorealistic bedroom”, “photorealistic kitchen”, “photorealistic living room”. This method augments the image with semantic backgrounds, aiming to closely resemble real-world scenarios.

GreenAug-Mask. This variant uses a masking (soft segmentation) network trained to predict masks. These predicted masks are then applied to the image observations to obtain blacked-out, dark backgrounds. This simplification of the visual input potentially helps the visuomotor policies to focus on the main elements of interest by eliminating background noise and distractions. During training, the masking network processes images against chroma-keyed backgrounds with random textures (akin to GreenAug-Rand) and learns to predict the masks generated through chroma keying.

Table 1: Main experiment results averaged across three novel scenes. Each task-method combination is evaluated with 112 evaluation episodes on average. Full detailed results are provided in the Appendix.

	Success Rate (%)
Task	NoAug	CVAug	Generative Augmentation	GreenAug Rand	GreenAug Gen	GreenAug Mask
Open Drawer	59	65	77	96	87	79
Place Cube in Drawer	36	69	70	92	83	37
Take Lid off Saucepan	67	81	77	88	73	71
Sweep Coffee Beans	66	78	75	96	81	84
Place Jeans in Basket	71	75	76	87	77	67
Place Bear in Basket	45	63	61	95	49	41
Stack Cups	49	59	77	81	72	55
Slide Book and Pick Up	49	74	89	93	93	35
Average	55	70	75	91	77	58

3.3 Training Robot Learning Policies

GreenAug can be applied to RGB-based robot learning methods. Similar to standard augmentation methods, images can be transformed with GreenAug and fed into policy networks during training, or they can be preprocessed offline and then used for training. Offline preprocessing is more common due to the longer computation time of some GreenAug variants. However, in online settings such as reinforcement learning, online transformations are also effective. GreenAug-Rand and GreenAug-Gen allow each raw frame from the training demonstrations to be augmented with different textures, significantly increasing the amount of preprocessed data. In contrast, GreenAug-Mask only masks the background and provides a single solution. To ensure a fair comparison, we keep the number of preprocessed frames equal to the number of raw frames for all methods.

In our main experiment (Section 4.3), we chose Action Chunking with Transformers (ACT) [4] as our control variable to demonstrate the effectiveness of this augmentation method. We selected ACT because of its recent success in adapting behaviours from a modest number of demonstrations, making it an ideal platform to showcase the benefits of GreenAug. Additionally, in Section 4.4, we demonstrate that GreenAug-Rand is also effective with a reinforcement learning policy.

4 Experiments

In this section, we present the experiments to evaluate the effectiveness of GreenAug on robot learning policies. Prior works [20, 39] have confirmed the effectiveness of background and texture randomisation in simulation. Since GreenAug focuses on real-world data augmentation, our experiments are conducted exclusively in the real world. We aim to study the following: (1) Does GreenAug improve visual generalization to unseen scenes? (2) Which variant of GreenAug is the most effective, and what are the tradeoffs? (3) Is GreenAug applicable in different data collection settings? (4) Is GreenAug agnostic to robot embodiments and learning methods?

4.1 Baselines

We implement several baselines to compare with GreenAug, as described below.

No augmentation (NoAug). No visual augmentation.

Computer Vision augmentation (CVAug). Random photometric distortions and random shift.

Generative augmentation. Generative augmentation encompasses a broader range of methods such as CACTI [8], GenAug [9], and ROSIE [10]. CACTI uses Stable Diffusion for inpainting but does not detail the method for obtaining object masks. GenAug, on the other hand, is constrained to a tabletop setting. ROSIE relies on proprietary models and does provide publicly available code. Thus, we have developed our own implementation that closely aligns with these methods. Our implementation is based on Grounding DINO [40] for open vocabulary object detection, Segment Anything [41] for zero-shot segmentation, and Stable Diffusion Turbo [12, 13] for inpainting, integrated with ControlNet [42] and conditioned on DPT-Hybrid [43] (monocular depth estimator) for better generation. Generative augmentation is similar to GreenAug-Gen, but it uses object detection and segmentation for mask creation instead of chroma keying. The pseudocode detailing this implementation is outlined in the Appendix.

4.2 Setup

Figure 4: Visualisations of train and test scenes.

For our main experiment, we designed eight tasks (illustrated in Fig. 7) and structured our experiments for each task as follows.

Data collection. We collected two sets of demonstrations, each consisting of 50 demos. One set was recorded against a green screen (Scene 1), and the other within a standard setting (Scene 2). All data were collected using a leader-follower teleoperation system, similar to ALOHA [4], but with a 7-DoF Franka Panda arms and a 2F-140 Robotiq gripper on the follower. We used three D415 Realsense cameras, positioned at the upper wrist, lower wrist and left shoulder camera. The images are captured at a resolution of 240 (height) x 320 (width) pixels. For the main experiments alone, we collected over 800 demonstrations and conducted more than 6.6k evaluation runs. Additionally, we gathered about 50 more training demonstrations and 1.6k evaluations for the ablation and further studies in Section 4.4.

Training. We trained all baselines and our methods on both sets of data, except for GreenAug, which was excluded from Scene 2 as it relies on the green screen. Each data set corresponds to a separate policy. ACT is used as the control policy for our main experiments.

Evaluation. In addition to Scenes 1 and 2, we evaluated the methods in three novel scenes (Scenes 3–5). Initially, each method was assessed in Scene 1 to establish an upper-bound performance for the task. Subsequently, the methods were evaluated in Scene 3–5 to test generalisation. For each combination of task, method, train scenes (2), test scenes (3), we performed 25 evaluation runs.

Each scene is shown in Fig. 4. To focus on testing visual generalisation across different scenes, we maintained the positions and orientations of the objects (while applying the same degree of randomisation for one-to-one comparison) relative to the robot while moving between scenes.

Table 2: Processing time per RGB frame.

Method Time ($\downarrow$) Generative Augmentation 2.530 s GreenAug-Rand 0.009 s GreenAug-Gen 0.882 s

Table 3: GreenAug-Rand applied to RL.

Success Rate (%) Train Scene Test Scene NoAug GreenAug Rand Green Screen 1 Novel Scene 12 64

Table 4: GreenAug-Rand with different texture types averaged across tasks and novel scenes. Entropy signifies the amount of texture randomness.

Texture Type Entropy (bits) Success Rate (%) None - 48 Solid Colours 0.00 65 Perlin Noise 4.45 66 MIL Textures 6.81 87

Table 5: Object generalisation results. Policies trained on a green cup were tested on other objects. (n) specifies the number of objects tested in the category.

Success Rate (%) Object Category NoAug GreenAug Rand GreenAug Gen Cups (3) 95 83 80 Cans (2) 38 46 40 Cubes (2) 0 32 52 Soft Toy (1) 0 84 72 Average 45 61 62

4.3 Results

Figure 5: Visualisations of raw and preprocessed frames (left shoulder and lower wrist camera views) of generative augmentation, GreenAug-Gen and GreenAug-Mask (during inference). Both generative methods struggle with producing good contextual wrist camera inpainting. In generative augmentation, the gripper is inpainted as part of the background, while GreenAug-Mask shows masking artefacts in novel scenes.

Table 1 presents our experimental findings. The results demonstrate that GreenAug-Rand surpasses all other baseline methods across all tasks. Specifically, GreenAug-Rand shows approximately a 65% improvement over NoAug, around a 29% improvement compared to CVAug, and about a 21% improvement over generative augmentation.

Surprisingly, GreenAug-Gen and generative augmentation rank second and third in performance respectively, despite using semantically meaningful backgrounds like living rooms or kitchens. As expected, both methods perform similarly, since they differ only in how they obtain background masks (object detection and segmentation). This suggests that specific semantic content is not crucial for GreenAug’s success, as the variant using random backgrounds performs even better. This superior performance may have resulted from the wider variety of colours and textures offered by the random backgrounds.

Generative augmentation performs slightly worse than GreenAug-Gen, likely because it struggles to provide good masks in wrist camera views (illustrated in Fig. 5), which are essential for tasks requiring precise and stable visual input. Despite advancements in generative models, segmentation and inpainting from robot camera views remain suboptimal.

GreenAug-Mask shows the least effectiveness among all methods tested. Qualitative evaluations of the masked images reveal frequent failures to completely obscure backgrounds, especially in novel scenes (shown in Fig. 5). This issue stems from two main factors: the inherent imperfections in ground truth masks obtained from chroma keying and the compounding error from the masking network. The network’s imperfect masking further complicates the tasks, pushing the images into out-of-distribution states that challenge the control policy.

4.4 Ablation and Further Studies

((a))

((b))

Figure 6: (6(a)) GreenAug-Rand performs the best when applied to all frames per trajectory. (6(b)) Visual assessment of applying GreenAug to a single shade of green, in scenarios where multiple objects with varying shades of green are present. Masked objects have their backgrounds blacked out.

Based on the main experiments, we demonstrated that GreenAug-Rand outperforms all other methods. We then conducted the following in-depth analyses.

Benchmarking GreenAug’s speed. We conducted a benchmark to compare the processing speed of various methods, shown in Table 5. CVAug and GreenAug-Mask were excluded because the former is applied on the fly during training, and the latter performs poorly. We show that GreenAug-Rand is significantly faster than the other two generative methods.

Applying GreenAug to a different robot with reinforcement learning. We investigated whether GreenAug can be applied to a different robot embodiment and learning method, beyond the Franka Panda and ACT. We set up a similar “take lid off saucepan” task on a UR5. We used a continuous demo-driven DQN variant [44, 45, 46, 47] with actions discretised into bins. The robot was provided with 24 demonstrations and was given a sparse reward of 0 for failure and 1 for success. We trained the robot online with 20 minutes of exploration on a green screen background and evaluated two policies, NoAug and GreenAug-Rand, in one novel scene. The results, shown in Table 5 demonstrate that GreenAug-Rand applied to reinforcement learning with a different robot performs significantly better than NoAug.

Impact of texture randomness. We investigated how the texture randomness of GreenAug-Rand affects performance. We tested solid colours, Perlin noise (procedurally generated textures) [48], and MIL textures (used in the main experiments). All texture datasets are of the same size (5771). The evaluation was conducted on the “put cube in drawer” and “stack cups” tasks from the main experiment across three novel scenes (Scenes 3–5). Table 5 summarises the results. Consistent with domain randomization studies [25, 26, 27, 28, 29, 31], greater texture randomness leads to better performance. Examples of each texture type are provided in the Appendix.

Generalisation across object category. We assessed if GreenAug can be applied not just to backgrounds but also to different object categories. We set up a simple pick-and-place task. We first trained on a green cup and then tested on other visually different objects. The results, shown in Table 5, indicate that GreenAug-Gen performs best, with only a 1% difference from GreenAug-Rand. Both methods outperform NoAug by more than 35%. NoAug performs well on cups but fails with cubes and soft toys, and occasionally works with cans due to their similar geometric shapes to cups. GreenAug-Rand and GreenAug-Gen show better performance across different object categories, demonstrating some level of generalisation. However, performance with cups suffers slightly, likely due to the strong augmentation causing confusion about geometric shapes.

Green screen coverage. In real-world settings, some frames in the robot data may move away from the green screen during robot servoing. For example, if the green screen is only partially set up in the scene, the robot may observe parts of the scene not covered by the green screen. To emulate this scenario, we applied GreenAug-Rand to varying percentages of frames per episode. This was evaluated on the same tasks as the texture randomness study. The results are summarised in Fig. 6(a). As expected, green screen coverage is proportional to the success rate.

Presence of multiple green objects. Green screens could affect scenes when there are multiple green objects. We evaluated the sensitivity of chroma keying under these conditions, a challenge also encountered in the film industry. This study questions whether chroma keying can effectively isolate one green object without impacting others. We conducted a visual assessment (shown in Fig. 6(b)) and showed that we can augment only one object at a time while leaving the others unchanged. Alternatively, one can also use a different colour such as blue (along with a green background) for chroma-keying objects.

5 Conclusion and Limitations

Figure 7: Visualisations of real-world tasks. The trajectory sequences are stacked horizontally, starting with the initial positions labelled as #0.

This paper proposes and investigates the efficacy of GreenAug in robotic manipulation across a variety of real-world scenarios. We have demonstrated that GreenAug not only works effectively across different tasks but also surpasses other augmentation methods in performance while maintaining simplicity. GreenAug outperforms NoAug by approximately 65%, CVAug by 29% and generative augmentation by about 21%. Our findings advocate for a paradigm shift in data collection practices for robot learning. We propose the use of green screens for future real-world demonstrations. Implementing GreenAug could significantly improve policy generalisation across novel locations, effectively addressing scene generalisation limitations currently faced in the field.

While GreenAug proves to be useful, several challenges remain that we have outlined for future research. GreenAug is effective for background generalisation and to an extent, object generalisation (as shown in further studies), but it falls short when it comes to adapting to objects with very different geometric shapes. This type of generalisation involves changing the dynamics and trajectories of the demonstrations, such as accommodating different mugs with unique handles that require specific grasping points. Furthermore, GreenAug could be complementary to generative augmentation. This combination could help train world models capable of producing imaginary trajectories that generalise across diverse objects and appliances.

Acknowledgments

Big thanks to the members of the Dyson Robot Learning Lab for discussions and infrastructure help: Nic Backshall, Iain Haughton, Younggyo Seo, Sridhar Sola, Jafar Uruc, Yunfan Lu, Abdi Abdinur, Nikita Chernyadev.

Appendix A Experiment Setups

In this section, we provide the detailed setups of our real-robot experiments to help reproduce the results.

Robot Setup. The robot setup consists of a 7-DoF Franka Panda Emika arm equipped with a Robotiq 2F-140 gripper. We use three RealSense D415 cameras: two cameras mounted on the end-effector (lower wrist, upper wrist) for a wide field-of-view, and one camera (left shoulder) fixed on the base, as depicted in Fig. 8(a).

Data collection. We gather demonstrations for our tasks utilising a leader-follower setup similar to ALOHA [4]. An expert human demonstrator moves the Leader arm, and the Follower arm mirrors the Leader’s joint positions, as shown in Fig. 8(b). Camera and robot state observations are recorded at 30 FPS.

Tasks. For each task, we collect 50 demonstrations each at two scenes: green screen room and living room. Fig. 11 shows the task definitions with sketches to illustrate the setup with measurements and randomisation. For all tasks, the initial robot joint positions are [0.0, -0.785, 0.0, -2.356, 0.0, 1.571, 0.0].

((a)) Robot setup. Franka Panda Arm is mounted on a Vention base with three Realsense cameras. The robot is above the ground by 23. The robot pose is represented in the base frame, ${F_{R}}$.

((b)) Leader and follower robot setup.

((a)) Open Drawer: The robot base ($F_{R}$) is uniformly randomised inside the $\qty{10}{}\times\qty{10}{}$ region. The gripper then slides into the small drawer opening and then pulls the drawer open. In total, 50 demonstrations are collected with an average demo length of 169 steps or 13 secs.

((b)) Place Cube in Drawer: The robot base ($F_{R}$) is fixed relative to the drawer. The cube is randomised on the drawer top within the $\qty{32}{}\times\qty{10}{}$ region. The robot picks up the cube and places it inside the opened drawer. A total of 50 demonstrations are collected with an average demo length of 250 steps or 19 secs.

((c)) Sweep Coffee Beans: The robot base ($F_{R}$) is fixed relative to the drawer. The drawer used is the same as in previous tasks but rotated by 90. We stick two black tapes on the drawer top. The coffee beans are randomised in the $\qty{24}{}\times\qty{24}{}$ region between the two tapes. The sponge position is randomised along the tape B (24). The robot grasps the sponge and sweeps the coffee beans to the left of tape A. A total of 50 demonstrations are collected with an average demo length of 314 steps or 24 secs. For evaluations, a trial is considered successful if 90% of beans are swept. We use 20 beans, so at least 18 beans needs to be swept for success.

((a)) Take Lid off Saucepan: The robot base ($F_{R}$) is fixed relative to the drawer. The saucepan is randomized in the L-shaped region. The robot grasps the lid of the saucepan and always places it in the orange area. In total, 50 demonstrations were collected with an average demo length of 857 steps.

((b)) Place Jeans to Basket: The robot base ($F_{R}$) is fixed relative to the laundry basket. The jeans is semi-folded and hanging on the right edge of the chair. The chair position is randomised in the $\qty{20}{}\times\qty{20}{}$ region. The robot grasps the jeans from the side and places it in the laundry basket. We collected 50 demonstrations, in each half of the demonstrations the chair position is randomised keeping the orientation A and in the other half the orientation of the chair remained B. The average length of the demo is 592 steps or 44 secs.

((c)) Place Bear in Basket: The robot base ($F_{R}$) is fixed and the bear (toy) position is randomised in the $\qty{46}{}\times\qty{30}{}$ region on the ground. For half of the demos, the randomisation is done in orientation A and for the other half in orientation B. The robot first picks up the toy and places it in the basket nearby. We collect 50 demonstrations in total with an average demo length of 741 steps or 56 secs.

((a)) Stack Cups: The robot base ($F_{R}$) is fixed relative to the drawer. The orange cup is randomised in the $\qty{10}{}\times\qty{32}{}$ region whereas the blue cup is randomised along the blue line (32cm). The robot first picks up the orange cup and stacks it on the blue cup. In total, 50 demonstrations are collected with an average demo length of 590 steps or 44 secs.

((b)) Slide Book and Pick Up: The robot base ($F_{R}$) is fixed relative to the black coffee table. The book position is randomised in the $\qty{12}{cm}\times\qty{12}{cm}$ region. In half of the demonstrations, the book orientation is kept A and in the other half, orientation B. The robot first corrects the book orientation if necessary by pushing on its edge and then slides the book to the edge of the table. It then picks it up and places it in the area depicted by orange (rightmost figure). We collect 50 demonstrations in total with an average demo length of 930 steps or 70 secs.

((c)) Place Cup in Drawer (Object Generalisation): The robot base ($F_{R}$) is fixed and the green cup position is randomised in the $\qty{20}{cm}\times\qty{32}{cm}$ region on the drawer top. The robot first picks up the cup and places it in the drawer. We collect 50 demonstrations in total with an average demo length of 597 steps or 45 secs.

Figure 11: Task definitions with randomisation. We illustrate 8 main tasks and 1 ablation task with randomisation used. We used the images from the left shoulder (top row) and lower wrist camera (bottom row) to describe each task sequence. Note that the sketches on the right are not drawn to scale.

Figure 12: Reinforcement learning experiment setup (“Take the lid off saucepan”). The illustration shows train and test scenes and the task sequence. In the train scene, a green screen cloth covers the table. In the test scene, the cloth is removed, and distractors are added around the table. UR5 robots are used for this experiment, with only upper and lower wrist cameras.

Appendix B More Visualisations

Figure 13: Visual observations of GreenAug-Rand with MIL textures [49]

Figure 14: Visual observations of GreenAug-Rand with solid & Perlin textures

Figure 15: Visual observations of GreenAug-Gen

Figure 16: Visual observations of GreenAug-Mask during inference.

Figure 17: Visuals of three novel scenes used during evaluations for each task in respective order. These include a subset of kitchen, washing, study, and living rooms.

Appendix C Compute and Hyperparameter Details

We perform the preprocessing and model training using NVIDIA L4 GPUs (24GB VRAM).

ACT. We use the same implementation of ACT as described in the original paper, with the following changes to hyperparameters: action chunking size is set to 20, the number of epochs is 5000, and we sample 16 transitions per epoch. Unlike the original ACT implementation, which samples one transition per episode per epoch, we sample multiple transitions.

Table 6: Pre-processing hyperparameters for each task. Chroma key parameters are represented by Key Colour ($K$) in hexadecimal colour codes, tola ($\alpha$), and tolb ($\beta$). Detection Text Prompt is used for generative augmentation. Inpaint Text Prompt is used for both generative augmentation and GreenAug-Gen for background generation.

Task	Key Colour ($K$)	tola ($\alpha$)	tolb ($\beta$)	Detection Text Prompt	Inpaint Text Prompt
Open Drawer	#439f82	30	35	drawer. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Place Cube in Drawer	#25806f	35	40	red cube. drawer. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Sweep Coffee Beans	#1d6953	23	30	sponge. coffee beans. black tapes. drawer. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Take Lid off Saucepan	#348367	15	25	saucepan. drawer. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Place Jeans in Basket	#25806f	30	40	jeans. chair. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Place Bear in Basket	#25806f	30	30	soft toy. basket. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Stack Cups	#348367	15	25	blue cup. orange cup. drawer. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Slide Book and Pick Up	#25806f	20	30	book. table. robot arm. robot gripper.	photorealistic kitchen, study room, washing room, living room, or bedroom
Object Generalisation	#699230	30	20	green cup. drawer. robot arm. robot gripper.	colourful cup, bowl, cube, toy, can, bottle or general graspable object

GreenAug-Mask U-Net. We use the original U-Net architecture [50] (implemented by Iakubovskii [51]) for the masking network used in GreenAug-Mask. The model comprises 14.3 million parameters.

Table 7: Masking network hyperparameters for GreenAug-Mask.

Model	Unet
Encoder	ResNet18
Encoder Weights	ImageNet
Epochs	100
Batch size	128
Image size	$224\times 224$
Seed	42

Appendix D Detailed Results

This section presents the full unaggregated results.

Table 8: Full experiment results. “Green Screen→Green Screen” roughly represents the upper bound performance (in parentheses) and is not included in the average. Full unaggregated results for each task are in Tables 9, 10, 11, 12, 13, 14, 16 and 15. The tables are also hyperlinked in the task text below.

			Success Rate (%)
Task	Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Open Drawer	Green Screen	Green Screen	(100)	(88)	(96)	(100)	(100)	(100)
	Living Room	3 Novel Scenes	63	51	57	-	-	-
	Green Screen	3 Novel Scenes	55	79	96	96	87	79
Place Cube in Drawer	Green Screen	Green Screen	(92)	(96)	(72)	(100)	(84)	(96)
	Living Room	3 Novel Scenes	33	64	68	-	-	-
	Green Screen	3 Novel Scenes	39	73	72	92	83	37
Sweep Coffee Beans	Green Screen	Green Screen	(100)	(96)	(88)	(96)	(80)	(92)
	Living Room	3 Novel Scenes	55	79	73	-	-	-
	Green Screen	3 Novel Scenes	77	77	77	96	81	84
Take Lid off Saucepan	Green Screen	Green Screen	(96)	(84)	(92)	(80)	(80)	(84)
	Living Room	3 Novel Scenes	61	79	72	-	-	-
	Green Screen	3 Novel Scenes	73	83	83	88	73	71
Place Jeans in Basket	Green Screen	Green Screen	(100)	(100)	(92)	(100)	(100)	(92)
	Living Room	3 Novel Scenes	69	73	75	-	-	-
	Green Screen	3 Novel Scenes	72	77	77	87	77	67
Place Bear in Basket	Green Screen	Green Screen	(100)	(100)	(100)	100	(96)	(100)
	Living Room	3 Novel Scenes	35	45	41	-	-	-
	Green Screen	3 Novel Scenes	55	80	81	95	49	41
Stack Cups	Green Screen	Green Screen	(76)	(84)	(84)	(88)	(80)	(80)
	Living Room	3 Novel Scenes	41	57	75	-	-	-
	Green Screen	3 Novel Scenes	57	61	80	81	72	55
Slide Book and Pick Up	Green Screen	Green Screen	(100)	(100)	(100)	(100)	(100)	(100)
	Living Room	3 Novel Scenes	43	61	89	-	-	-
	Green Screen	3 Novel Scenes	55	87	89	93	93	35
Average			55	70	75	91	77	58

Table 9: “Open Drawer” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(100)	(88)	(96)	(100)	(100)	(100)
Living Room	Novel Scene 1	60	48	68	-	-	-
	Novel Scene 2	68	64	64	-	-	-
	Novel Scene 3	60	40	40	-	-	-
Green Screen	Novel Scene 1	52	88	88	100	84	76
	Novel Scene 2	72	80	100	100	96	80
	Novel Scene 3	40	68	100	88	80	80

Table 10: “Place Cube in Drawer” task unaggregated results..

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(92)	(96)	(72)	(100)	(84)	(96)
Living Room	Novel Scene 1	68	88	72	-	-	-
	Novel Scene 2	0	32	64	-	-	-
	Novel Scene 3	32	72	68	-	-	-
Green Screen	Novel Scene 1	36	92	76	92	92	48
	Novel Scene 2	56	68	64	96	92	48
	Novel Scene 3	24	60	76	88	64	16

Table 11: “Sweep Coffee Beans” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(100)	(96)	(88)	(96)	(80)	(92)
Living Room	Novel Scene 1	60	96	88	-	-	-
	Novel Scene 2	52	60	88	-	-	-
	Novel Scene 3	52	80	44	-	-	-
Green Screen	Novel Scene 1	92	80	88	100	92	96
	Novel Scene 2	76	80	88	96	80	76
	Novel Scene 3	64	72	56	92	72	80

Table 12: “Take Lid off Saucepan” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(96)	(84)	(92)	(80)	(80)	(84)
Living Room	Novel Scene 1	64	76	76	-	-	-
	Novel Scene 2	68	84	68	-	-	-
	Novel Scene 3	52	76	72	-	-	-
Green Screen	Novel Scene 1	64	80	80	88	76	68
	Novel Scene 2	96	96	92	84	76	80
	Novel Scene 3	60	72	76	92	68	64

Table 13: “Place Jeans in Basket” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(100)	(100)	(92)	(100)	(100)	(92)
Living Room	Novel Scene 1	64	64	68	-	-	-
	Novel Scene 2	76	76	76	-	-	-
	Novel Scene 3	68	80	80	-	-	-
Green Screen	Novel Scene 1	68	76	76	84	72	64
	Novel Scene 2	72	80	76	80	80	72
	Novel Scene 3	76	76	80	96	80	64

Table 14: “Place Bear in Basket” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(100)	(100)	(100)	(100)	(96)	(100)
Living Room	Novel Scene 1	32	24	32	-	-	-
	Novel Scene 2	36	24	12	-	-	-
	Novel Scene 3	36	88	80	-	-	-
Green Screen	Novel Scene 1	56	92	80	100	44	32
	Novel Scene 2	28	52	68	84	12	24
	Novel Scene 3	80	96	96	100	92	68

Table 15: “Stack Cups” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(76)	(84)	(84)	(88)	(80)	(80)
Living Room	Novel Scene 1	44	52	64	-	-	-
	Novel Scene 2	40	60	76	-	-	-
	Novel Scene 3	40	60	84	-	-	-
Green Screen	Novel Scene 1	56	44	76	72	60	24
	Novel Scene 2	64	80	88	84	72	80
	Novel Scene 3	52	60	76	88	84	60

Table 16: “Slide Book and Pick Up” task unaggregated results.

		Success Rate (%)
Train Scene	Test Scene	NoAug	CVAug	Generative Augmentation	GreenAug random	GreenAug generative	GreenAug mask
Green Screen	Green Screen	(100)	(100)	(100)	(100)	(100)	(100)
Living Room	Novel Scene 1	68	84	88	-	-	-
	Novel Scene 2	28	48	84	-	-	-
	Novel Scene 3	32	52	96	-	-	-
Green Screen	Novel Scene 1	44	96	92	96	96	32
	Novel Scene 2	48	72	84	88	88	8
	Novel Scene 3	72	92	92	96	96	64

Table 17: Texture randomness unaggregated results (GreenAug-Rand). “Green Screen→Green Screen” roughly represents the upper bound performance (in parentheses) and is not included in the average.

			Success Rate (%)
Task	Train Scene	Test Scene	None	Solid Textures	Perlin Textures	MIL Textures
Place Cube in Drawer	Green Screen	Green Screen	(92)	(100)	(100)	(100)
	Green Screen	Novel Scene 1	36	68	64	92
		Novel Scene 2	56	36	56	96
		Novel Scene 3	24	68	68	88
Stack Cups	Green Screen	Green Screen	(76)	(76)	(80)	(88)
	Green Screen	Novel Scene 1	56	76	68	72
		Novel Scene 2	64	68	60	84
		Novel Scene 3	52	76	80	88
Average			48	65	66	87

Table 18: Green screen coverage unaggregated results (GreenAug-Rand). “Green Screen→Green Screen” roughly represents the upper bound performance (in parentheses) and is not included in the average.

			Success Rate (%)
Task	Train Scene	Test Scene	0%	25%	50%	75%	100%
Place Cube in Drawer	Green Screen	Green Screen	(92)	(100)	(100)	(100)	(100)
	Green Screen	Novel Scene 1	36	80	76	80	92
		Novel Scene 2	56	64	68	72	96
		Novel Scene 3	24	88	84	88	88
Stack Cups	Green Screen	Green Screen	(76)	(76)	(80)	(76)	(88)
	Green Screen	Novel Scene 1	56	64	76	80	72
		Novel Scene 2	64	68	60	68	84
		Novel Scene 3	52	72	76	76	88
Average			48	73	73	77	87

Table 19: Object generalisation unaggregated results. Data is collected on the green cup, and policies are then trained and evaluated on various objects (illustrated in Fig. 18).

	Success Rate (%)
Object Type	NoAug	GreenAug-Rand	GreenAug-Gen
Green Cup	96	88	80
Blue Cup	96	80	80
Orange Cup	92	80	80
Red Cube	0	44	60
Green Cube	0	20	44
Soda Can	40	28	36
Soya Can	36	64	44
Soft Toy	0	84	72
Average	45	61	62

((a)) Demonstrations collected using the green cup.

((b)) Test objects: cups, cans, cubes and toy.

Figure 18: Object generalisation across object category. Policy trained on green cup data using GreenAug-Rand and tested on different objects.

Appendix E Additional Limitations and Future Works

Exploration of better chroma key algorithms. The chroma key algorithm used in this paper [38] is a basic one that performs reasonably well, but it does not produce perfect masks. Some parameter tuning for $K$, $\alpha$, and $\beta$ is still necessary. Despite these imperfections, we demonstrate that GreenAug still significantly outperforms the baselines. In the film industry, extensive manual post-processing is often required to achieve perfect masks [52]. Future research could explore more advanced chroma key algorithms that provide superior green screen masks [17, 53, 54, 18]. This could potentially enhance the performance of GreenAug-Mask, which relies heavily on green screen mask as ground truth for training.

Pose generalisation. A major ongoing challenge in robot learning is generalising to 6D poses not present in the training dataset. Current robot learning policies, especially imitation learning-based ones often fail when objects are relocated to different positions within 3D space.

Application to methods with 3D observations. Currently, GreenAug has only been tested on RGB-based robot learning policies. Recent advances in next-best-pose-based agents [55, 3, 6] have demonstrated that by aligning the observation space with action space, we can obtain strong generalisation in robot learning policies. As a general plug-and-play method, GreenAug could potentially further improve the scene generalisation of the next-best-pose agents, which we leave for future study.