Installation and Migration

Installation

Requirements

It is recommended to run Refine with:

  • At least 1 gigabyte of memory.
  • At least 256 megabytes of disk space.
  • Java Runtime Environment or Java SE Development Kit versions between 11 and 15
    • The recommended JRE is Temurin-11.0.15+10.
    • Java is not required for native installations, which include that JRE
    • We recommend using Refine with 64-bit version of Java

Note

Refine is an in-memory application, meaning that it loads the full data of the current project in memory. Disk storage is needed for input files, autosave of the project and its metadata (every minute), and exported files.

If you have large data, you should provide more memory. We can define large data as:

  • More than 800,000 total cells.
  • An input size of more than 50 megabytes.
  • More than 50 rows per record.
Number of cells Minimal memory Optimal memory
400,000 512MB 1GB
800,000 1.5GB 3GB
2,000,000 3GB 6GB
4,000,000 6GB 12GB
6,000,000 8GB 16GB
8,000,000 10GB 20GB

As a rule of thumb, optimal memory grows by 2GB for each million cells.

You can run Refine with lower memory, but you are risking an Out of Memory (OOM) exception and we can’t recommend such a deployment.

Warning

Resource usage is cumulative for all projects that are created or opened in this session. For example, if you try to open two projects, one with 2,000,000 cells and another with 4,000,000 cells, you should have 8GB of memory.

If you use the cross() function in a project, the other project is also opened. This function is useful for combining datasets between Refine projects.

Running Refine

Refine can be operated as a desktop or a server application. Choose the one that best suits your needs and follow the steps below to run Refine:

  • As a desktop (native) installation: For desktop users who want to test out the application. It is very straightforward, sets up an uninstaller, and desktop shortcuts.
  • As a standalone server: Recommended for production. Comes packaged with a preconfigured web server. It offers greater configurability.
  • In a Docker container: A stable way to have an installation that “just works”, without having to worry about the specifics of your operating system. See our images on Dockerhub.

Native Launchers

The easiest way to set up and run Refine is to use the native desktop installations. This does not require the use of a console and works with a graphic user interface. For this installation approach, you do not need to download Java, as the application comes with JDK 11 bundled.

Go to the Refine download page and request your copy. You will receive an email with the download link. According to your OS, proceed as follows:

Windows

  1. Download the Refine .msi installer file.
  2. Double-click the application file and follow the on-screen installer prompts.
  3. You will be asked for an installation location.
  4. Locate the application in the Windows Start menu or on the desktop and start it.
  5. The Refine application opens at http://localhost:7333/.

MacOS

  1. Download the Refine .dmg file.
  2. Double-click it and get a virtual disk on your desktop.
  3. Copy the program from the virtual disk to your hard disk Applications folder.
  4. Locate the application on the desktop and start it.
  5. The Refine application opens at http://localhost:7333/.

Linux

  1. Download the Refine .deb or .rpm file.
  2. Install the package:
  • Debian and derivatives: sudo dpkg -i <package-name>.deb
  • Redhat and derivatives: sudo rpm -i  <package-name>.rpm
  • CentOS and derivatives: sudo yum install <package-name>
  1. Alternatively, double-click the package.
  2. Locate the application on the desktop and start it.
  3. The Refine application opens at http://localhost:7333/.

Configuring Native Refine

Native Refine uses two configuration files:

  • {ontorefine.dist}/Ontotext Refine.cfg are Java launcher properties. See JVM Configuration for details.
    • On Windows, this is \\Users\\<username>\\AppData\\Local\\Ontotext Refine\\app\\Ontotext Refine.cfg
  • {user.home}/OntoRefine/ontorefine.properties are Refine properties. See Refine Configuration Parameters for details.
  • On Windows, this is \\Users\\<username>\\AppData\\Roaming\OntoRefine\ontorefine.properties
  • You can change this location using the Java property ontorefine.config.path or environment variable ONTOREFINE_CONFIG_PATH.

Once you have started the native application, a small icon appears in the statusbar/menu/tray area. It allows you to check if the application is running, stop it or change the configuration settings. Additionally, an application window is opened, where you can open the Refine web app, go to the Refine documentation, configure settings, and see all log files. You can reopen this window by choosing Show Refine window from the statusbar icon menu.

_images/nativeLauncher.png

In addition, a few of the parameters (for connecting to GraphDB) can be set in the Refine web app.

Stopping Native Refine

To stop the application, simply quit it from the statusbar/menu/tray area icon, or close the Refine application window.

Hint

On some Linux systems, there is no support for statusbar/menu/tray area. If you have hidden the Refine window, you can quit it by killing the process.

Standalone Server

The recommended way of running Refine in production use cases is as a standalone server. The server is platform-independent and includes all recommended JVM parameters for immediate use.

Note

The standalone distribution does not package a JDK like the native launcher. Before running Refine, please make sure to have JDK or JRE installed.

Running Standalone Refine

  1. Download the Refine distribution archive and unpack it.
  2. Start Refine by executing the ontorefine startup script located in the bin directory of the Refine distribution.

A message appears in the console telling you that Refine has been started. You can access the Refine client at http://localhost:7333/ in your browser.

Configuring Refine Server

See Refine Configuration Parameters for a description of Refine settings.

Docker

In addition to Standalone Server, Ontotext Refine is also packaged and distributed as a Docker image.

Quick Start

The following Docker Compose file can be used to quickly get Refine up and running.

version: "3"

services:
  refine:
    image: ontotext/refine
    restart: unless-stopped
    environment:
      SERVER_HOST: localhost
    ports:
      - "7333:7333"
    volumes:
      - refine_data:/opt/ontorefine/data

volumes:
  refine_data:

After saving the Docker Compose file, start it by executing the next shell command in the same folder as the file:

docker-compose up -d

Refine is accessible by default at http://localhost:7333/.

Note

It is advisable to update the SERVER_HOST environment variable to the actual hostname.

Configuring Docker

You can pass any of the configurations parameters described below as environment variables in uppercase convention. For example, server.host should be written as SERVER_HOST.

Refine Configuration Parameters

You can assign configuration parameters in three ways:

  • Edit the configuration file.
  • Set corresponding environment variables. E.g. the property server.host should be written as SERVER_HOST.
  • Add command-line options to the startup script. You can also pass JVM options that way. For example, to change the config file location, Refine port number, and set Java max heap, use:
ontorefine -Dontorefine.config.path=<your-config-file> -Dserver.port=<your-port> -Xmx4g

If the same parameter is set in multiple locations, the following order of precedence applies: config file < environment variables < command-line arguments.

Refine takes the following configuration parameters:

  • ontorefine.config.path: Location of Refine config file. Created on first startup of Native Refine. Can be changed using the Java command-line property ontorefine.config.path or environment variable ONTOREFINE_CONFIG_PATH, but not within the config file itself. Defaults:

    • On Mac: {user.home}/.ontorefine/ontorefine.properties
    • On Windows: \\Users\\<username>\\AppData\\Roaming\OntoRefine\ontorefine.properties
    • On Linux: {user.home}/.ontorefine/ontorefine.properties
  • ontorefine.data: Location of Refine data (project files and workspace.json). Defaults:

    • On Mac: {user.home}/.ontorefine
    • On Windows: \\Users\\<username>\\AppData\\Roaming\OntoRefine\\refine
    • On Linux: {user.home}/.ontorefine
  • ontorefine.dist: Refine installation directory. Can be set during installation of Native Refine. Should not be edited, unless Refine is moved manually. Defaults:

    • On Mac: /Applications/Ontotext Refine.app/Contents/app
    • On Windows: \\Users\\<username>\\AppData\\Local\\Ontotext Refine\\app
    • On Linux: /opt/ontorefine/lib/app/
  • server.host: server on which Refine runs. Useful for CORS configurations. Default: your machine’s hostname

  • server.port: port on which the Refine server responds. Default: 7333

  • graphdb.address: URL address of the GraphDB instance that Refine asks for namespaces and class/property autocompletion. Default in the Setup menu of the web app: http://localhost:7200

  • graphdb.repository: ID of the GraphDB repository that Refine works with. Default in the Setup menu of the web app: refine

  • graphdb.auth.method: Authentication method used for GraphDB Access Control. Case-insensitive. Default: NONE. Possible settings:

    • NONE: used only when GraphDB security is OFF. This should only be used on a local network not connected to the internet.

    • BASIC: the username and password are sent in a header as plain text (see Basic authentication). Sending the password as plain text is insecure, so it is important to use Encryption in Transit (i.e. https) to improve security.

      • graphdb.auth.basic.username: username for Basic authentication.
      • graphdb.auth.basic.credentials: password for Basic authentication.
    • GDB: token-based authentication, also used by GraphDB Workbench (see GDB authentication). Refine uses the GraphDB login API method to obtain a token: this happens on Refine startup, and whenever the token expires. Token expiration is controlled by the graphdb.auth.token.validity parameter in GraphDB (default: 30d). Refine uses the following parameters to perform GraphDB login:

      • graphdb.auth.gdb.username: username for GDB authentication.
      • graphdb.auth.gdb.credentials: password for GDB authentication.

Important

You can also connect a GraphDB instance using the visual dialog in Refine’s “Setup” section.

_images/install-connect-gdb.png

JVM Configuration

You can use the following environment variables to control Java startup parameters:

  • ONTOREFINE_JAVA_OPTS: Sets additional Java parameters (-D or -X)
  • ONTOREFINE_JAVA_32BIT: If set to true, configures Refine to run on 32bit Java
  • ONTOREFINE_MIN_MEM: Sets the Java minimum heap size (-Xms option).
  • ONTOREFINE_MAX_MEM: Sets the Java maximum heap size (-Xmx option).
  • ONTOREFINE_HEAP_SIZE: Sets the Java minimum and maximum heap size (-Xms and -Xmx option).
  • ONTOREFINE_HEAP_NEWSIZE: Sets the initial and maximum heap size for the young generation (-Xmn option).
  • ONTOREFINE_GC_LOG: If set to true, enables the logging of Java garbage collection.
  • ONTOREFINE_GC_LOG_FILE: Specifies a custom file for logging of Java garbage collection.

Other Java options are defined at the end of the file {ontorefine.dist}/Ontotext Refine.cfg. Each java-options= line provides a single argument passed to the JVM when it starts (-cp means “class path”).

Here are the options with their default settings on Windows:

[JavaOptions]
java-options=-Djpackage.app-version=1.0
java-options=-cp
java-options=$APPDIR\ontorefine-native-app.jar;$APPDIR\lib\*
java-options=-Dfile.encoding=UTF-8
java-options=-Dontorefine.dist=$APPDIR
java-options=--add-exports
java-options=jdk.management.agent/jdk.internal.agent=ALL-UNNAMED
java-options=--add-opens

java-options=java.base/java.lang=ALL-UNNAMED

It is recommended not to remove or change any of the existing options provided with the installation. You can add your own options at the end.

For example, if you want to run Refine with 8 Gb of maximum heap memory, you can set the following option:

java-options=-Xmx8g

The launcher spawns the Refine application within the same JVM that it operates in, so you cannot change JVM options from the command line.

Migrating Existing Projects from GraphDB 9

Migrating projects from existing GraphDB instances to Refine is possible in bulk or manually.

Manual Migration

You can use manual migration for a small number of projects:

  • Export the projects one by one from GraphDB 9 using the “Export project” function
  • Import the projects one by one to Refine 1.0 using the “Import project” function

Bulk Migration

Bulk Migration migrates all projects from a Refine workspace at once.

  • Make sure that the old OntoRefine (part of GraphDB 9) is not being used

  • There are several ways in which you can transfer the workspace (set of projects):

    1. Set the {ontorefine.data} configuration parameter (of the new Refine) to point to the old data folder
    2. Replace the entire contents of the new (Refine) data folder with the contents of the old data folder
    3. Move project subfolders from the old data folder to the new data folder. (If a project ID already exists in the new data folder, leave it alone). Then edit the projectIDs array in workspace.json to list all projects.
    • This alternative is for advanced users only
    • Make sure you keep a backup of the old and new workspace folders
  • Restart Refine in order to reload the projects