Our crawler is developed using Selenium. We use a Selenium built-in function driver.installAddon(PATH_TO_EXTENSION) to install our analysis extension on the automated browser. This function only takes an Firefox add-on in xpi format. To pack the extension into xpi format, follow the below instructions precisely:

1. Download 7zip here.

2. Navigate to the gpc-web-crawler/gpc-analysis-extension directory. Open a terminal and build the project by running:

   npm run build

3. Navigate to the gpc-web-crawler/gpc-analysis-extension/dist/firefox directory. Find manifest.json and open it. Scroll down to the bottom of the script. Find "incognito": "spanning". Add the below setting:

   "browser_specific_settings": {
       "gecko": {
         "id": "{daf44bf7-a45e-4450-979c-91cf07434c3d}"
       }
     }
   

   Your script should look like this:

   

   Save it!

4. Still under the gpc-web-crawler/gpc-analysis-extension/dist/firefox directory. On your terminal, run:

   7zz a -tzip myextension.xpi * -r -mx9

   You should see that myextension.xpi appears under the current directory. Move it to the gpc-web-crawler/selenium-optmeowt-crawler directory.

   Tip: if you are making a lot of modifications to the analysis extension and need to frequently pack it to test these changes, it is definitely worthwhile to use a shell script like this one (xpi-packing-shell-script.txt) to pack the extension and put it into the selenium-optmeowt-crawler directory for you.

5. To test if your extension is packed successfully, you can open a Firefox Nightly browser and directly drag myextension.xpi file into your browser window. If a pop-up window appears and asks if you want to install the add-on, you have successfully converted your extension into xpi format. If the pop-up window tells you the extension appears to be corrupt, make sure you have done all the above steps correctly. Check if you performed each step under the correct directory. If the pop-up window tells you the extension is unsigned, check Step 6.

6. The extension you just packed into xpi format can only work in a Firefox Nightly browser. Open a Firefox Nightly browser and navigate to about:config. Search xpinstall.signatures.required and set it to false. Try to install your xpi extension again. If the pop-up window still tells you the extension is unsigned, check if you did Step 3 correctly.