Generating PBS Files

xFDS can programmatically generate .pbs files!

If you need .pbs files, simply add the .pbs file to the files parameter in the config file and build your .pbs file as you would any other template.

In the simple atrium example, a variable proc is defined to indicate the number of processors required and is accessable in the template.

In Python, // is the operator for floor division. Therefore, {{ proc // <ppn> }} will produce the number of full nodes required. For example, if the model requires 28 nodes and each node has 8 processors, {{ proc // 8 }} will render as 3 implying that 3 full nodes are required.

In Python, % is the operator for determining the remainder. Therefore {{ proc % <ppn> }} will produce the number of cores needed on a partial node. For example, if the model requires 28 nodes and each node has 8 processors, {{ proc % 8 }} will render as 4 implying only 4 cores are required on that node.

xfds:
  render:
    - name: atrium_{{cfm}}_{{mua_perc}}
      files:
        - simple_atrium.fds
        - simple_atrium.pbs
      variables:
        proc: 1
...

#PBS -l nodes={{ proc // 12 }}:node1:ppn=12:1:node1:ppn={{ proc % 12 }}
#PBS -l nodes+={{ proc // 16 }}:node2:ppn=16:1:node2:ppn={{ proc % 16 }}
#PBS -l nodes++={{ proc // 20 }}:node3:ppn=20:1:node3:ppn={{ proc % 20 }}
#PBS -N {{ name }}
#PBS -S /bin/bash
#PBS -m abe
#PBS -M xfds@pbd.tools
#PBS -j oe

# load required modules
module load fds/6.7.5

# change to the working directory
cd $PBS_O_WORKDIR

# set MPI variables
export I_MPI_PIN=1
export I_MPI_PIN_MODE=pm
export OMP_NUM_THREADS=1
export MPI_PPN=$(($PBS_NUM_PPN / $OMP_NUM_THREADS))
export MPI_NP=$(($PBS_NP / $OMP_NUM_THREADS))

# run fds in parallel
mpiexec -np $MPI_NP fds_mpi {{ name }}.fds

Tip

Use the xfds.render.parameters.include directive if you need to modify the number of processors based on the a specific configuration.

If you require fewer cores than available on a single node, {{ n // 8 }} will render to 0. Similarly, if the number of required cores is a multiple of the cores on a single node, {{ n % 8 }} will render to 0. If this is undesireable, use an if statement containing the node filter.

The node filter requires two parameters in addition to the number of cores needed: ppn, or how many processors there are per node, and what mode to consider. The modes are as follows:

• full: Returns True if at least one full node is required.
• part: Returns True if a partial node is required.
• both: Returns True if both full and part are required.

Tip

Use Jinja's whitespace control syntax to collapse multiple lines onto one line.

xfds:
  render:
    - name: using_{{n}}_nodes
      files:
        - model.fds
        - model.pbs
      parameters:
        n: [4, 8, 12]

#PBS -l nodes=
{%- if n|node(ppn=8, mode='full') %}{{ n // 8 }}:node:ppn=8{% endif %}
{%- if n|node(ppn=8, mode='both') %}:{% endif %}
{%- if n|node(ppn=8, mode='part') %}1:node:ppn={{ n % 8 }}{% endif %}

If only 4 of the 8 processors on a node are needed, only 4 are requested.

#PBS -l nodes=1:node:ppn=4

If all 8 of the 8 processors on a node are needed, all 8 are requested.

#PBS -l nodes=1:node:ppn=8

If 12 processors are needed, but there are only 8 processors on a node, a full node of 8 processors and a partial node of 4 processors are requested.

#PBS -l nodes=1:node:ppn=8:1:node:ppn=4