This feature allows running multiple IncrediBuild executions in parallel on the same Initiator machine.
Users can execute parallel Visual Studio compilations, BuildConsole commands, IBConsole commands or any combination of the above.
Support for parallel execution of multiple IncrediBuild executions is off by default, in order to turn this feature on, navigate to the IncrediBuild Agent Settings -> Initiator -> Multiple Builds and select the configuration that best suits your needs.
Scenarios relevant for concurrent IncrediBuild executions
This feature is especially useful in the following scenarios:
- Concurrent builds being executed from a build server - when several developers working against a single build server or in continuous integration and continuous delivery cycles, it is common to require builds to run concurrently, especially in scenarios where more than one developer issues a build to be executed on a build machine at the same time.
- Executing multiple solutions in parallel - there are various scenarios, especially in C# development, where multiple C# solutions can be executed in parallel. Being able to execute multiple such solutions in parallel in conjunction with IncrediBuild’s capability of distributing the compilation tasks of solution’s projects to remote machines can highly accelerate the overall time it takes to execute the full build cycle.
- There are various scenarios in which executing jobs in parallel with IncrediBuild can highly reduce the overall time of the build cycle. For example, running unit tests and integration tests in parallel or generating assets while executing post build steps.
- In game development, it is common for developers to execute other compute-intensive jobs with IncrediBuild (such as shader compilations, rendering, etc.) in parallel to executing compilation tasks. This parallel build execution feature, allows developers to execute shader compilations and at the same time continue developing and compiling their source code while the compilation of the shaders is taking place.
Precautions when running concurrent IncrediBuild executions
Some precautions need to be taken into account when running multiple concurrent IncrediBuild executions on the same initiating machine:
- Make sure that the initiating machine can handle multiple job executions without getting over-provisioned in terms of resources, especially memory consumption and I/O access. For example, with IncrediBuild, a link task will only be executed on the local machine. Executing too many links in parallel on a machine without sufficient memory resources can lead to out of memory issues or result in high swapping that may cause a negative impact on performance.
- If multiple builds are to be executed in parallel, make sure these builds do not write to the same files or folders.
- It is recommended to experiment with enabling the “Avoid task execution on local machine when possible” flag under the IncrediBuild Agent Settings -> Initiator -> General section. Enabling this flag instructs IncrediBuild to prefer executing tasks on remote machines when possible, making the initiating machine’s resources available to execute tasks that can only execute locally and leave as much of the machine’s resources free to service the remote machine’s synchronization requests.
Limiting the number of concurrent IncrediBuild executions
In order to allow users to limit the number of IncrediBuild concurrent executions on an Initiator machine according to the resources available on that machine, two new flags were added to Agent Settings -> Initiator -> Multiple Builds. The maximum number of IncrediBuild commands that can run in parallel is capped by these two values:
Limit number of concurrent builds to (default is 1, meaning only a single build will be allowed to execute each time) - in order to be able to run concurrent IncrediBuild executions, either specify a value higher than 1, or simply uncheck this option, so there won’t be a limit to the number of concurrent IncrediBuild executions you can run.
Note that the number of concurrent builds is still capped by the number of cores licensed to this Agent and the value set to the “minimum number of cores per build” flag(see below).
Minimum local cores per build (default is 4) - controls the minimum number of local cores that must be available (i.e. not occupied by another IncrediBuild execution) in order for another IncrediBuild execution to start concurrently.
Figure 1: Agent Settings flags that control the concurrent IncrediBuild executions feature
For example:
In a machine with 10 local cores in which the number concurrent build was limited to 3, with the “Minimum local cores per build” flag set to 4:
- The first IncrediBuild execution, will use all of the 10 local cores (along with additional Helper cores).
- When the second IncrediBuild execution starts, the 10 local cores will be split evenly between the 2 IncrediBuild executions and each will use 5 local cores (along with additional Helper cores), which meets the requirement of minimum number of 4 local cores per build. Both builds will run concurrently.
- The third build will not start and will be added to the IncrediBuild execution queue, as it won’t have enough local cores to start in parallel with the other executions: If the 10 local cores will be split between the 3 builds (4 + 4 + 2), the last build will only have 2 local cores which is not enough to start a new build, so it will have to wait until one of the currently running builds will finish and release its used local cores.
Note that the first build is not affected by the value of minimum local cores per build.
e.g. if the machine only has two cores and the minimum required number is set to 4, it is still possible to run a single build, that will only use two cores.