GSoC Final Report: GNU Radio Hardware-in-the-Loop Controller for CortexLab

Contributor: [Sayantan Maity]

EMail: [maitysayantan116@gmail.com]

Organization: [GNU Radio]
Mentor(s): [Cyrille Morin, Marcus Müller]
Project period: June-August 2026
Repository: GitHub Link

Abstract

This project delivers a Python-based controller for repeatable GNU Radio hardware-in-the-loop (HIL) experiments on CortexLab. The controller turns a multi-step manual workflow—reserving radio nodes, starting a CortexLab MINUS task, staging experiment files, coordinating transmitter and receiver programs, and evaluating results—into a single HTTP request with observable status throughout the run.

The implementation provides a Flask API and web dashboard, OAR reservation and MINUS task orchestration, gateway-mediated SSH/SFTP deployment, synchronized multi-node execution, experiment/result registries, and two reusable GNU Radio experiment examples: a basic over-the-air tone test and an OFDM payload test. It also includes an early container-image build path for testing GNU Radio revisions. The project establishes the controller and experiment interface needed for eventual physical-hardware CI, while clearly leaving full CI integration, durable state, authentication, and additional end-to-end validation as future work.

Motivation

GNU Radio software that works in local simulation still needs validation against real SDR hardware. RF behavior depends on radio drivers, timing, gain and frequency settings, the physical RF environment, and coordination between distributed transmitter and receiver nodes. CortexLab offers remotely reservable SDR nodes, but a manual test requires several systems to be operated in the correct order.

The goal was therefore to provide a small, reusable HIL control plane that can run an experiment from a structured request and expose its progress to a developer or, later, to CI. The intended workflow is:

  1. Reserve compatible CortexLab nodes through OAR.
  2. Create a MINUS task using a GNU Radio-capable container image.
  3. Determine which allocated nodes are online and available.
  4. Create a per-run workspace, upload scripts and parameters, and launch the node programs together.
  5. Collect a machine-readable result from an analysis program.

Completed Work

Controller API and dashboard

I implemented a Flask controller that starts locally on port 5678. Its main POST /run-experiment endpoint accepts reservation data, experiment selection, pull-request metadata, and parameters. It either creates or reuses a reservation, waits for the reserved nodes to become available, then delegates the run to the generic experiment runner. The project also exposes status endpoints for reservations, nodes, all experiments, and an individual experiment; the browser dashboard refreshes this information every two seconds.

This makes the state of a run inspectable rather than leaving users to infer progress from remote shell sessions. The execution identifier follows <pr-id>-<commit-prefix>-<oar-job-id>, enabling a hardware run to be traced back to the revision that requested it.

Relevant implementation: flask_server.py, templates/dashboard.html, and experiment_manager/generic_experiment_runner.py.

Reservation, task, and node lifecycle

The CortexLab integration submits and monitors OAR reservations and creates MINUS scenarios/tasks for them. A background reservation monitor queries oarstat -fj, records reservation state, derives scheduled wait time, and parses assigned CortexLab hostnames. Node monitoring updates the in-memory registry with online/offline state and association with an experiment.

The generic runner selects only online, non-busy nodes, verifies that enough nodes exist for the requested experiment, and marks selected nodes busy before staging. It releases them after primary node execution is complete. This avoids assigning a node already in use by another tracked run.

Relevant implementation: cortexlab/reservation/, cortexlab/nodes/, and cortexlab/remote_connections/.

Reusable experiment packaging and remote deployment

An experiment is defined as a directory containing node_scripts/, analysis.py, and an example parameter file. Every Python script in node_scripts/ is mapped to one available radio node. The controller creates an isolated local folder for each run, copies the selected scripts and analysis program into it, writes parameters.json, and uploads the workspace to the CortexLab reservation area over SSH/SFTP.

This layout separates controller behavior from experiment-specific GNU Radio programs. Adding a new experiment does not require changing the orchestration path: the experiment author provides node scripts, parameters, and analysis logic following the same contract.

Relevant implementation: experiment_manager/create_experiment_folder.py, experiment_manager/upload_experiment_folder.py, and experiment_manager/hil_experiments/.

Synchronized distributed execution and result handling

The controller launches one thread per node script. A threading.Barrier and a shared future start time coordinate scripts so that the transmitter and receiver begin together rather than serially. Execution state, start/end timestamps, standard output/error, and results are recorded per node and at experiment level.

After every primary node finishes successfully, the controller runs analysis.py. Analysis is expected to produce a JSON result with status, reason, and metrics; a result whose status is passed is treated as a successful experiment. Failures from node execution, analysis, malformed result output, or insufficient nodes are recorded in the experiment registry.

Relevant implementation: experiment_manager/start_experiment.py, cortexlab/execution/, and cortexlab/execution/execute_analysis.py.

HIL experiment implementations

Two example experiment families are included.

  • Basic hardware test: a transmitter and receiver run a tone-based SDR test. The analysis reads captured complex IQ samples, checks received sample count, finite samples, average signal power, and dominant-frequency error. It writes metrics such as RMS amplitude, frequency resolution, tolerance, and pass/fail checks to results.json.
  • OFDM hardware test: GNU Radio transmitter and receiver flowgraphs, together with parameterized Python scripts, exercise an OFDM payload path. The analysis compares the received binary payload with the expected UTF-8 message and provides byte-level diagnostics and a machine-readable verdict. The experiment has been executed on CortexLab, but the expected payload is currently not being written to the output file and therefore requires further debugging.

The accompanying .grc flowgraphs, node scripts, analysis scripts, and parameter.json files make these examples useful both as smoke tests and as templates for future experiments.

Relevant implementation: experiment_manager/hil_experiments/basic_hardware_test/ and experiment_manager/hil_experiments/ofdm_hardware_test/.

Initial CI and GNU Radio Image Groundwork

The repository includes a Dockerfile for building a CortexLab GNU Radio 3.10 environment with the required UHD, VOLK, GNU Radio, and additional GNU Radio modules. The Dockerfile is adapted from the CorteXlab/cxlb-docker-gnuradio-3.10 Dockerfile and extends its setup to accept a configurable GNU Radio Git reference, including a specific commit SHA. This allows an image to be built from the exact GNU Radio revision associated with a pull request.

This work intentionally stops short of claiming complete CI integration. No checked-in CI workflow currently builds and publishes an image from a PR SHA, passes the resulting image reference through the experiment request to MINUS, runs a HIL experiment, and reports the result back to the originating pull request. The current scenario path still uses the configured default image at run time.

Relevant implementation: ci_workflow/docker_image/Dockerfile.pr.

Acknowledgement: The base Dockerfile and CortexLab GNU Radio container setup were adapted from the CorteXlab/cxlb-docker-gnuradio-3.10 project.

Architecture

Developer / CI request
          |
          v
Flask API + Dashboard
          |
          +-- OAR reservation monitor --> assigned CortexLab nodes
          |
          +-- MINUS scenario/task --> GNU Radio container image
          |
          +-- Experiment manager --> per-run folder + parameters.json
          |                              |
          |                              v
          |                         SSH/SFTP staging
          |                              |
          v                              v
Execution registry <--- synchronized node scripts (TX/RX)
          |
          v
analysis.py --> results.json --> final status and metrics

Development Timeline

The repository history shows continuous development from June through late August 2026. The work progressed from remote SSH access, OAR reservation and scenario generation, and a reservation dashboard, through task creation, node monitoring, file upload, execution tracking, synchronized execution groups, analysis/result collection, and finally the basic and OFDM experiment implementations plus image-build parameterization.

Notable milestones include:

  • June: remote gateway SSH workflow, OAR reservation/scenario generation, initial Flask status UI, and early tests.
  • Early July: reservation/task monitoring, node state monitoring, remote file upload, dashboard improvements, and execution logging/tracking.
  • Mid to late July: execution state model and synchronized execution groups.
  • Early August: result downloading, result normalization, and dashboard status improvements.
  • Mid to late August: basic hardware analysis, OFDM TX/RX implementation, parameterized GNU Radio image building, and CortexLab node-configuration updates.

Testing and Validation

The basic_hardware_test has been successfully tested end-to-end on CortexLab. The test covers hardware reservation, scenario generation, experiment execution, and result collection, and it completed successfully.

The OFDM test has also been executed on the hardware. Currently, there are no significant errors reported in the execution logs, but the expected message is still not being written to the output file. I started debugging the issue step by step, but due to the current CortexLab infrastructure issues, I could not continue the debugging further.

Therefore, the basic_hardware_test is confirmed to be working successfully, while the OFDM test requires further debugging once the CortexLab issues are resolved. The next step is to continue investigating the OFDM RX/data path and verify why the received message is not being written to the output file.

Challenges and Lessons Learned

The main challenges were related to coordinating experiments across shared, remote CortexLab hardware. Reservation is asynchronous, so a submitted job does not immediately mean that the assigned nodes are ready for execution. I also faced issues with node connectivity through the gateway, file transfer, experiment execution, reservation monitoring, and synchronization between transmitter and receiver nodes. These issues highlighted the importance of handling reservation, node, and execution states explicitly and monitoring them throughout the experiment lifecycle.

During development, I also encountered several integration and debugging issues, including Python import and packaging problems, Flask/controller endpoint issues, reservation-state handling, remote SSH connections, and MINUS task submission. Debugging the OFDM experiment was particularly challenging because the TX/RX flowgraph could execute without producing obvious runtime errors, while the expected received message was still missing from the output file. I was able to successfully test the basic_hardware_test, but further OFDM debugging was temporarily blocked by CortexLab infrastructure issues.

Another important lesson was that a practical HIL framework needs a clear and consistent experiment structure. Keeping experiment-specific scripts and configuration inside a standard folder structure while the controller manages reservation, file transfer, execution, synchronization, and result collection makes it easier to add new experiments.

Finally, real-hardware CI is a system-integration problem rather than only a Docker or CI workflow problem. The complete system must account for the GNU Radio version being tested, Docker image creation and publishing, MINUS scenario generation, CortexLab hardware availability, authentication and secrets, reservation and cleanup, experiment results, and reporting the final status back to the originating pull request.

Remaining Work

The main remaining tasks are:

  • Integrate a production CI workflow that builds and publishes the requested GNU Radio image, passes its image reference to MINUS scenario generation, runs the controller, and reports the verdict back to the originating pull request.
  • Add more tests to cover all hardware-relevant blocks.
  • Refine the multi-experiment mechanism to properly handle cases where a task is already running.
  • Maintain a record of which tests cover particular blocks and add a mechanism to request tests by block name.
  • Improve failure cleanup, retries, timeouts, and artifact-retention policies for long-running shared-hardware jobs.
  • Continue debugging and complete the OFDM payload test once the CortexLab infrastructure is stable.

Conclusion

The project has delivered the essential control path for GNU Radio HIL testing on CortexLab: one request can reserve hardware, stage a parameterized experiment, start coordinated radio programs, collect analysis artifacts, and expose a final machine-readable result.

The successfully validated basic_hardware_test demonstrates the end-to-end HIL workflow on physical CortexLab hardware. The OFDM experiment establishes the payload-level testing and analysis workflow, but its final payload verification still requires further debugging.

The resulting codebase is a credible foundation for physical-hardware regression testing of GNU Radio changes. Completing CI integration, resolving the remaining OFDM issue, and adding production hardening will turn that foundation into a more repeatable, secure, and maintainable HIL validation service.

Acknowledgements

Before starting this project, I only had a basic understanding of Python. I would like to sincerely thank my mentors Cyrille Morin and Marcus Müller for their continuous guidance and support throughout the project. With their help, I learned and worked with GNU Radio, CortexLab, Flask, Paramiko, Docker, and Docker image building, and gained a much deeper understanding of hardware-in-the-loop testing and CI.

I am also grateful to the CortexLab operators for providing the infrastructure and support needed to run the hardware experiments, and to the GNU Radio community for their guidance, feedback, and valuable resources throughout the project.