Process Virtualization Technology

IncrediBuild accelerates time-consuming products or executions by distributing their processes in parallel across idle cores over the local network machines on which IncrediBuild is installed. For example, IncrediBuild enables an execution running on a four-core local machine, to run on 400 cores in the network, thus significantly accelerating the overall execution time.

IncrediBuild uses a unique technology to accelerate these processes and applications: Process Virtualization. This technology virtualizes processes on a remote machine as if they are running on the local machine.

Note:

• Process Virtualization is similar to VMware. VMware virtualizes processes on a complete remote computer that is set for that purpose in advance; Process Virtualization runs a process on demand on a remote machine.

Process Virtualization technology ensures that a Task is executed exactly as if it is being executed on the computer that initiated the Job (Initiating Machine), regardless of the remote Agent's file system, installation base, and environment.

Process Virtualization does not require any installations on remote agents, and does not require files to be copied from Initiating Machine. The only requirement is that the small and light IncrediBuild agent is installed on all machines. Process Virtualization ensures that files (including DLLs and others) required by the remote process are transferred automatically and transparently from Initiating Machine to Helper Machines.

Process Virtualization provides an additional feature – interception, which takes place on the local machine. If a parent process has to run many child processes, and wants to run these child processes on a remote machine, IncrediBuild knows to listen to all process executions of the parent process. With the help of a small XML file that defines the processes that should be run in remote, IncrediBuild knows to monitor all the process creations of the parent, and for each process that it creates, looks at a predefined XML file to see whether the process has been defined by the user to run remotely.

Process Virtualization technology is appropriate to all IncrediBuild solutions (VS, Make and Build Tools, and Dev Tools), with only small differences between them:

• Visual Studio, in addition to Process Virtualization technology, also has optimization for VS in the form of a plug-in, and added features that add optimization to VS compilation.
• For Make and Build Tools, IncrediBuild created a profile that has proved to work, and added technology to deal with issues that commonly occur with Make.
• For Dev tools, the same technology is used; however, the user does integration using IB interfaces.

Remote Virtualized Environment

Helper agents are client programs on various machines in the network that run parts (Tasks) of an executed Job in a virtual environment so that the results are the same as if the Tasks would be run on the local machine. Helper agents communicate with the Coordinator, reporting the availability of their resources (cores). As a result, the Coordinator is always up to date as to how much processing power each Helper agent can provide.

Note

• IncrediBuild runs on Helper computers without affecting the users working on the computers, as it only uses the computers' idle CPU cycles. The users are not aware of or disturbed by IncrediBuild running on their machines, as the users get their required resources, and IncrediBuild only uses the computers' available resources.

When a Task is assigned to execute on a Helper agent via IncrediBuild, a process representing that Task runs on the Helper machine in a special virtualized environment. Any child processes that are run by this process are also virtualized processes.

IncrediBuild fully emulates the initiating machine's environment for the Helper agent, including: file system, registry, process DLLs, standard output, and directory information. All remotely-performed Tasks run in this encapsulated environment. There is no need to copy files from the original machine or install applications on remote machines. IncrediBuild supports interaction among all Windows-based agents on a network, even when they are running different versions of Windows operating systems.

Virtualized Processing Data Flow

The Helper executes the process and surrounds it with the virtualization environment, simulating the environment of the originating machine. In the Helper machine, each call that the process makes to the operating system is intercepted by the virtualization environment. If a call is related to the file system, the virtualization environment synchronizes the file from the initiating machine to the remote machine, and caches it there in a dedicated folder. The virtualization environment then replaces the original path (for example c:\a.txt) with the path to the cache (for example c:\IncrediBuild\cache\a.txt). The same logic applies to registry access, DLL loading, etc.

At the end of the process, the Initiator machine gets the agent back as available, and decides whether it wants to run another Task on it, or wants to free it to the Coordinator.

For the host software running on the Initiating machine (for example, Visual Studio), this entire process is completely transparent (i.e., it appears as if the process was handled by the local operating system). In the Helper machine, the Task is handled by IncrediBuild on the OS as a virtualized process.

If the Helper machine is also being used by a conventional user, the additional (IB-initiated) processing is not apparent to the user. Most of the time, computers use very little of their CPU power. IncrediBuild makes use of much of this available CPU time that isn't in ongoing use by computers in the organization. The remote (virtualized) processes executed on the Helper machine only use the idle CPU cycles of the machine. In a medium-sized organization, this means that at any given moment, IncrediBuild can use hundreds of unused cores to accelerate processes in the organization. Each workstation in your organization that can initiate processes can serve as a workstation with the power of 200 cores instead of only 8 or 16 cores!

As the Helper machine does not have any of the related resources that are relevant to completing the Task, such as files or DLLs, the IncrediBuild Agent runs the process on the Helper and surrounds it with a virtual environment, requesting the required resources from the Initiator as it processes the task. These resources are stored in the Helper’s IncrediBuild file cache, so that the next time the process runs in the virtualized environment, the Helper can use the resources previously stored in its cache (assuming that they have not been modified since the last execution).

IncrediBuild has a simple set of interfaces that enables you to either directly ask to run a command on a remote Helper, or to define in advance names of processes, which upon detection, IncrediBuild runs automatically on a remote computer, provided one is available.

For example, when IncrediBuild accelerates builds in Visual Studio, it does so by adding the cl.exe process, the compiler of Visual Studio, and C++, to the list of predefined process names to be virtualized. This way, each time Visual Studio runs the cl.exe process, IncrediBuild automatically runs this process on remote computers. See Automatic Interception Interface.

If you run a batch file, which in turn runs many testing processes, you can specify that all processes running in the batch file are to run in parallel and on remote machines. See Submission Interface.

IncrediBuild's Virtualization Technology:

User Manual: