First, What is a Test?
A test is a procedure intended to establish the **quality, performance, or reliability** of something. It's a structured process of asking a system a question and observing its response to see if it matches our expectations.
Think of it like a chef tasting a sauce before serving it. They have an expected flavor (the requirement), and the taste (the test) confirms if it's right.
The Three Core Classes of Tests
Functional Test
The Question: "Can it do it?"
Verifies **what** the system does. It checks if a specific function meets its requirements, resulting in a simple pass/fail. It's about functionality, not quality.
Integration Test
The Question: "Do the parts work together?"
Verifies how different parts of a system work **together**. It focuses on the communication and data flow between components or subsystems.
Behavioral Test
The Question: "How well does it do it?"
Evaluates **how well** the system performs. It focuses on quality, efficiency, reliability, and performance under various conditions, not just a simple pass/fail.
System Architecture Levels
1. Component 🧩
The smallest, single, testable part of a system. The fundamental building block.
Ex: A motor, a sensor, a single software function.
2. Subsystem ⚙️
A collection of components that work together to perform a specific, larger task.
Ex: A complete drive train (motors, wheels, controllers).
3. System 🤖
The complete, final product, made by integrating all its subsystems into a self-contained entity.
Ex: A single, fully assembled robot.
4. System of Systems (SoS) 🌐
A collection of independent systems that collaborate to achieve a goal none could achieve alone.
Ex: A fleet of robots coordinating in a warehouse.
Test Applicability Matrix
| System Level | Functional Test | Integration Test | Behavioral Test |
|---|---|---|---|
| Component 🧩 | ✅ Primary Focus Ex: Sensor gives correct reading. |
❌ Not Applicable | ❌ Not Applicable |
| Subsystem ⚙️ | ✔️ Applicable Ex: Drive system moves on command. |
✅ Primary Focus Ex: Controller drives motors correctly. |
✔️ Applicable Ex: Lifter moves smoothly. |
| System 🤖 | ✔️ Applicable Ex: Robot completes A-to-B trip. |
✔️ Applicable Ex: Sensors correctly stop the drive system. |
✅ Primary Focus Ex: Robot recovers from a lost line. |
| System of Systems 🌐 | ✔️ Applicable Ex: Fleet moves 500 bins/hr. |
✅ Primary Focus Ex: Robots avoid collisions. |
✅ Primary Focus Ex: Fleet adapts to a robot failure. |
Visualizing the Framework
Framework Mind Map
PlantUML Code
You can use the code below in any PlantUML editor to generate or modify this diagram.
@startmindmap
' PlantUML Mindmap for General System Test Framework
<style>
mindmapDiagram {
.green { BackgroundColor lightgreen }
.yellow { BackgroundColor lightyellow }
.grey { BackgroundColor lightgrey }
.blue { BackgroundColor lightblue }
node { padding 8 }
}
</style>
* **General Testing Framework**
** **Component Level 🧩** <<blue>>
***_ **Functional Test** <<green>>
**** **Goal:** Verify the component's specific job (Unit Test).
**** **Example:** Sensor returns a value within its specified tolerance.
***_ **Integration Test** <<grey>>
**** **(Not Applicable)** Components are tested in isolation.
***_ **Behavioral Test** <<grey>>
**** **(Not Applicable)** Behavior emerges from interactions, not a single part.
** **Subsystem Level ⚙️** <<blue>>
***_ **Functional Test** <<yellow>>
**** **Goal:** Verify the subsystem's main function.
**** **Example:** Drive subsystem moves forward on command.
***_ **Integration Test** <<green>>
**** **Goal:** Verify components *within* the subsystem work together.
**** **Example:** Motor controller correctly drives the motors.
***_ **Behavioral Test** <<yellow>>
**** **Goal:** Check the quality/characteristics of the subsystem's operation.
**** **Example:** Lifter moves up and down smoothly.
** **System Level 🤖** <<blue>>
***_ **Functional Test** <<yellow>>
**** **Goal:** Verify the complete system's end-to-end functions.
**** **Example:** Robot successfully completes a trip from point A to B.
***_ **Integration Test** <<yellow>>
**** **Goal:** Verify all *subsystems* work together seamlessly.
**** **Example:** Sensor subsystem correctly signals the drive subsystem to stop.
***_ **Behavioral Test** <<green>>
**** **Goal:** Assess the overall quality, reliability, and intelligence.
**** **Example:** Robot recovers gracefully after losing the line it was following.
** **System of Systems (SoS) Level 🌐** <<blue>>
***_ **Functional Test** <<yellow>>
**** **Goal:** Verify the entire fleet achieves its collective objectives.
**** **Example:** Fleet transports 500 bins per hour.
***_ **Integration Test** <<green>>
**** **Goal:** Verify independent systems communicate and coordinate.
**** **Example:** Two robots successfully negotiate an intersection to avoid collision.
***_ **Behavioral Test** <<green>>
**** **Goal:** Assess the **emergent behavior** of the entire group.
**** **Example:** Fleet traffic re-routes efficiently when one robot is removed.
@endmindmapCase Study: Testing a Fleet of Autonomous Warehouse Robots
The Critical First Step: Architecture
Before writing a single test, a clear **system architecture** is essential. You must first define what the components are, how they assemble into subsystems, and how those subsystems depend on each other. **Without a defined system architecture and dependency map, it would be impossible to define these test boundaries meaningfully.** You wouldn't know what to test in isolation (component), what to test together (subsystem), or how to verify the complete product (system).
Level 1: Component Testing 🧩
Here, we test each part on a workbench, completely isolated from the rest of the robot. The goal is to verify the manufacturer's specifications.
Functional Tests (Primary Focus)
- LiDAR Sensor: In a controlled lab environment, place an object at exactly 1m, 3m, and 5m. Does the sensor's data output report these distances with an accuracy of +/- 2cm? This confirms its basic function before we rely on it for navigation.
- Wheel Motor: Using a power supply and a tachometer, send a specific voltage signal to the motor. Does it spin at the commanded RPM as stated in its datasheet? This ensures the motor itself isn't faulty.
- Software Function: For a function `calculate_path(start, end, map)`, we write a unit test. We provide a fixed start point, a fixed end point, and a simple, known map. Does the function return the expected, optimal path array? This validates the algorithm's logic.
Level 2: Subsystem Testing ⚙️
We now assemble components into functional groups. The focus shifts from "does it work?" to "do they work *together*?"
Integration Tests (Primary Focus)
- Drive Subsystem: Connect the main computer, motor controller, and wheel motors. Send a software command like `move_forward(speed=0.5)`. Do the motors receive the correct signals from the controller and turn in unison? This tests the entire command chain.
- Navigation Subsystem: Mount the IMU, wheel encoders, and LiDAR sensor together. As you physically move the assembly, does the sensor fusion algorithm produce a single, stable location estimate? We can intentionally cover the LiDAR to see if the system gracefully relies on the IMU and encoders without crashing.
Behavioral Tests
- Lifting Mechanism: Place a 20kg weight (max load) on the lifter. Command it to move up and down 100 times. We measure for smoothness, speed consistency, and signs of motor strain to assess its quality and durability.
Level 3: System Testing 🤖
The robot is fully assembled. We are now testing its ability to perform complete, end-to-end missions and handle real-world complexity.
Behavioral Tests (Primary Focus)
- Dynamic Obstacle Avoidance: Have a person walk across the robot's path. Does the robot detect the person, slow to a smooth stop at a safe distance, wait for the person to pass, and then gracefully resume its mission? This tests its real-world safety and intelligence.
- Error Recovery: While the robot is navigating, physically block its intended path. Does it stop, recognize the path is blocked, and successfully calculate a new route to its destination? This tests its problem-solving ability.
Functional & Integration Tests
- Full Mission Test: Command the robot to perform a complete task: "Go to location A, pick up the bin, transport it to location B, and drop it off." Does it complete every step in the sequence successfully? This is a key functional test.
Level 4: System of Systems (SoS) Testing 🌐
We now deploy the entire fleet of 20 robots into the test warehouse. The focus is on their collective, collaborative, and emergent behaviors.
Integration & Behavioral Tests (Primary Focus)
- De-confliction (Integration): Send two robots on paths that will intersect at the same time. Do they communicate their intent, follow the pre-defined rule (e.g., robot with the lower ID number yields), and pass through the intersection without collision?
- Task Allocation (Integration): Give the fleet manager 5 tasks simultaneously. Does it correctly evaluate the position and status of all 20 robots and assign each task to the closest, most appropriate robot?
- Fleet Resilience (Behavioral): While the fleet is operating, power down one robot unexpectedly. Does the fleet manager recognize the failure, automatically re-assign that robot's task to another, and continue operating with minimal disruption to overall throughput?
- Emergent Behavior (Behavioral): Give the fleet a very high number of tasks to create a high-traffic scenario. Do the robots' individual pathfinding algorithms work together to create smooth, efficient traffic flow, or do they create gridlock and bottlenecks in main corridors? This is a behavior of the *group*, not any single robot.