kub-simulate

1. Overview

kub-simulate launches and monitors feelpp_kub_cem jobs for a given location dataset version. It supports two launcher backends and two runtime backends:

  • slurm: submit via sbatch / srun

  • local: execute directly from the command line

  • native: run host binary directly

  • apptainer: run feelpp_kub_cem inside an Apptainer image

Default behavior is --launcher auto: use Slurm when available, otherwise fallback to local execution. Default runtime is --runtime native. It provides five operations:

  • preprocess: run mesh partitioning for one or more MPI partition counts

  • run: submit a simulation run with a generated run ID

  • resources: inspect local machine and Slurm cluster topology/capacity summaries

  • status: inspect recent runs and optional run details

  • config: manage persistent defaults/profiles in .kub-simulate.toml

2. Prerequisites

  • A valid dataset in cemdb/locations/<location>/…​ with <location>.cfg

  • For --launcher=slurm: sbatch and srun

  • For --launcher=local with --np > 1: mpiexec or mpirun

  • For --runtime=apptainer: an Apptainer executable and a local SIF image path (--apptainer-image)

  • squeue optional for live queue state in status

  • feelpp_kub_cem binary (auto-detected in build/default/src/cpp/feelpp_kub_cem or set with --binary)

3. Synopsis

kub-simulate [-h] [-v] [--config PATH] {preprocess,run,resources,status,config} ...

4. Commands

4.1. preprocess

Submit a preprocessing job (--mode=preprocessing) to generate partitioned meshes. The Slurm output reports a per-partition outcome hint: generated, skipped (already available), or forced rebuild requested.

kub-simulate preprocess [OPTIONS] <location>

Options:

Option Description

--version VERSION

Dataset version. auto (default) resolves current first, then the latest available v*.

--profile PROFILE

Optional profile name from config file (defaults + profile values are applied unless CLI option is explicitly set)

--cemdb-root PATH

Locations root path (default: cemdb/locations)

--binary PATH

Path to feelpp_kub_cem binary

--partition NAME

Slurm partition when launcher is slurm (default: public)

--time-limit HH:MM:SS

Slurm time limit when launcher is slurm (default: 02:00:00)

--launcher {auto,slurm,local}

Execution launcher (default: auto: Slurm when available, else local)

--runtime {native,apptainer}

Execution runtime (default: native)

--apptainer-image PATH

SIF image path (required when --runtime apptainer)

--apptainer-exec PATH

Apptainer executable path (default: $APPTAINER, fallback /opt/apptainer/latest/bin/apptainer)

--apptainer-bind HOST:CONTAINER

Additional bind mount passed to apptainer exec -B (repeat option)

--apptainer-env KEY=VALUE

Additional environment variable passed to apptainer exec --env (repeat option)

--partitions N [N …​]

Partition counts to generate (default: 32 64 128)

--force

Force preprocessing rebuild (--cem.preprocessing.force-rebuild=1)

--dry-run

Print generated launcher command without executing

4.2. run

Submit a simulation job (--mode=simulate) and create a run directory under: <location-version>/simulations/single/run_YYYY-MM-DD_HH-MM-SS.

kub-simulate run [OPTIONS] <location>

Options:

Option Description

--version VERSION

Dataset version. auto (default) resolves current first, then the latest available v*.

--profile PROFILE

Optional profile name from config file (defaults + profile values are applied unless CLI option is explicitly set)

--cemdb-root PATH

Locations root path (default: cemdb/locations)

--binary PATH

Path to feelpp_kub_cem binary

--partition NAME

Slurm partition when launcher is slurm (default: public)

--time-limit HH:MM:SS

Slurm time limit when launcher is slurm (default: 02:00:00)

--launcher {auto,slurm,local}

Execution launcher (default: auto: Slurm when available, else local)

--runtime {native,apptainer}

Execution runtime (default: native)

--apptainer-image PATH

SIF image path (required when --runtime apptainer)

--apptainer-exec PATH

Apptainer executable path (default: $APPTAINER, fallback /opt/apptainer/latest/bin/apptainer)

--apptainer-bind HOST:CONTAINER

Additional bind mount passed to apptainer exec -B (repeat option)

--apptainer-env KEY=VALUE

Additional environment variable passed to apptainer exec --env (repeat option)

--apptainer-mpi-tcp

Optional local OpenMPI workaround: add TCP transport flags when using Apptainer with --np > 1 (off by default)

--apptainer-mpi-tcp-iface IFACE

Interface used by --apptainer-mpi-tcp (default: lo)

--np INT

MPI process count / Slurm tasks (default: 32). In Slurm mode, default layout is auto-derived from partition topology.

--nodes INT

Slurm node count. If omitted, auto-derived from --np and per-node topology.

--ntasks-per-node INT

Slurm task density per node. If omitted, auto-derived from --np and selected node count.

--cpus-per-task INT

Slurm CPUs per MPI task (default: 1). Used to validate per-node task density.

--time-stop INT

Override cem.instance.time.stop (seconds)

--dry-run

Print generated launcher command without executing

Auto resource behavior for Slurm run:

  • Default packing uses physical cores per node (conservative on hyperthreaded nodes).

  • Example: on a partition with 128 physical cores/node, --np 256 resolves to --nodes 2 --ntasks-per-node 128.

4.3. status

Show preprocessing availability and recent simulation runs for a location.

kub-simulate status [OPTIONS] <location>

Options:

Option Description

--version VERSION

Dataset version. auto (default) resolves current first, then the latest available v*.

--cemdb-root PATH

Locations root path (default: cemdb/locations)

--last INT

Number of most recent runs to show (default: 10)

--run-id ID

Print detailed information for one run

4.4. resources

Show local machine resources and/or Slurm cluster capacity. CPU values are reported as both logical threads (hardware threads) and physical cores. Partition entries include per-node ranges (logical-threads/node, physical-cores/node, threads/core, gpus/node) and a homogeneous=yes|no flag.

kub-simulate resources [OPTIONS]

Options:

Option Description

--scope {local,cluster,all}

Select resource scope (default: all)

--partition NAME

Optional partition to report conservative per-node topology details

--no-hwloc

Disable hwloc-based local topology detection and use fallback discovery

4.5. config

Manage persistent user defaults and named profiles.

kub-simulate config {init,show,set,unset} ...

Common usage:

# Initialize local config file
kub-simulate config init

# Show current config
kub-simulate config show

# Set defaults
kub-simulate config set version 0.1.0
kub-simulate config set cemdb_root cemdb/locations

# Set a profile (example: apptainer-slurm)
kub-simulate config set runtime apptainer --profile-target apptainer-slurm
kub-simulate config set launcher slurm --profile-target apptainer-slurm
kub-simulate config set partition public --profile-target apptainer-slurm
kub-simulate config set apptainer_exec /opt/apptainer/latest/bin/apptainer --profile-target apptainer-slurm
kub-simulate config set apptainer_image ./kub-master.sif --profile-target apptainer-slurm

5. Examples

# 1) Preview a preprocessing submission script
kub-simulate preprocess arz --partitions 32 64 128 --dry-run

# 2) Submit preprocessing to a specific Slurm partition
kub-simulate preprocess arz --launcher slurm --partition public --time-limit 01:30:00

# 3) Force preprocessing assets to rebuild
kub-simulate preprocess arz --force

# 4) Submit a simulation with 128 MPI ranks on Slurm
kub-simulate run arz --version 0.1.0 --np 128 --launcher slurm --time-limit 04:00:00

# 5) Submit 256 ranks with explicit Slurm layout
kub-simulate run arz --np 256 --launcher slurm --nodes 2 --ntasks-per-node 128

# 6) Run locally from command line (no Slurm submission)
kub-simulate run arz --np 8 --launcher local --time-stop 172800

# 7) Run with Apptainer runtime (local launcher)
kub-simulate run arz --launcher local --runtime apptainer --apptainer-image ./kub-master.sif --np 4

# 8) Run with Apptainer runtime (Slurm launcher)
kub-simulate run arz --launcher slurm --runtime apptainer --apptainer-image /shared/images/kub-master.sif --np 128

# 9) Optional: enable MPI TCP workaround for local Apptainer runs
kub-simulate run arz --launcher local --runtime apptainer --apptainer-image ./kub-master.sif --np 4 --apptainer-mpi-tcp

# 10) Initialize config and use a profile to shorten commands
kub-simulate config init
kub-simulate preprocess arz --profile apptainer-slurm --partitions 32 64 128
kub-simulate run arz --profile apptainer-slurm --np 128

# 11) Check latest runs
kub-simulate status arz --last 20

# 12) Inspect one run in detail
kub-simulate status arz --run-id run_2026-03-04_05-23-05

# 13) Show local and cluster resources
kub-simulate resources --scope all --partition public

6. Exit Codes

  • 0: command completed successfully

  • 1: failure (invalid location/version, missing binary, launcher command error, unexpected runtime error)

7. Related Tools