BA/PA - Closed-Loop Control and Camera-Encoder Sensor Fusion for Precise Arm Movement and Canvas Painting

Closed-Loop Control and Camera-Encoder Sensor Fusion for Precise Arm Movement and Canvas Painting

Kamera- und Enkoder-basierte Sensorfusion zur kaskadierten Regelung von Armbewegung und Strichführung eines Malroboters

Taks description

Context

This topic is part of the overarching project “Exploring the Multimodality of Arts with Robotics”.

The idea for this project arose from the Ligeti Center, founded in 2023, which promotes interdisciplinary collaboration between art, science, medicine, and technology. The current focus is on the development of a robot capable of interpreting music to create a work of art. Originally developed as part of the Applied Design Methodology for Mechatronics (ADMM) course, this project has evolved through the efforts of several students and has resulted in the design of a robotic arm, controlled in ROS, that can create paintings based on music interpretation. The robot’s brush can move across a canvas, adjust its pressure, and mix a variety of colors, all while responding to specific musical parameters such as BPM, loudness, and mood.

The project integrates mechatronics , music theory , and human artistic interpretation , exploring different aspects of musical interpretation, from machine learning to human gestures to specific parameters of the music, in order to control the robot. By comparing the different outputs and processes with those of humans, the aim is to gain a better understanding of human music interpretation.

Workshops and public events are planned, where the robot will perform live and engage audiences in exploring the intersection of art and science.

Ongoing improvements to the robot’s design, control, and machine learning amongst others will be carried out through student theses and collaborations with various labs.

Task Description

Currently, the painting robot operates without dynamic detection of the canvas surface or working area—the canvas boundaries are hard-coded, requiring the arm to be positioned manually before startup. Furthermore, position tracking operates without feedback; while external encoders were mounted mechanically, they are not yet electrically connected or integrated into the control loop. Combined with mechanical play (primarily within the stepper motor gearboxes and loosening arm joints), this results in accuracy loss during execution.

To achieve consistent painting execution, reliable stroke control, and color quality, the system requires an upgrade to a fully integrated closed-loop control system. The goal of this thesis/project is to establish robust closed-loop arm control and dynamic canvas surface detection by implementing sensor fusion between the external encoders and vision-based camera feedback.

Tasks & Key Requirements

System & Dynamic Assessment: Perform a detailed analysis of the current system's limitations (quantifying positioning drift, speed capabilities, communication delays, latency, packet loss, and physical play).
Mechatronic Optimization: Identify and execute mechanical improvements to mitigate physical inaccuracies.
Encoder Integration & Closed-Loop Control: Electrically integrate the existing external encoders into the control framework to transition from open-loop actuation to closed-loop position/speed feedback.
Camera-Based Canvas & Arm Tracking: Integrate camera feedback to automatically detect the canvas working area (eliminating manual pre-positioning) and track arm trajectory/stroke quality on the canvas in real time.
Sensor Fusion: Implement a sensor fusion algorithm (e.g., Kalman Filter) merging encoder data and vision feedback to achieve accurate dynamic positioning and stroke execution during painting.
(Optional) Brush Pressure Control: Improve the closed-loop control of the lifting mechanism. This may include evaluating and replacing the current pressure sensor.

The tasks can be adapted to fit the type of thesis or project.

Possible areas of work / skills

Control Systems (ROS2)
Sensor Fusion (e.g., Kalman Filtering)
Computer Vision (OpenCV / camera-based tracking)
Mechatronics & Hardware Diagnostics (encoder wiring, gearbox backlash mitigation, joint mechanics)
Electronics & Sensor Integration (Force/Pressure sensors, Encoders)
C++ / Python
English language is highly preferred

Desired starting date

May 2027 (supervisor unavailable before), end preferably before start of WiSe 27-28

Contact person:

@Ornella Tortorici

Name:
Thesis Type MA/BA/PA:
Student ID / Matrikelnummer:
Field of Study / Studiengang:
Official start-date / Offizieller Beginn:
Final-report-due /Abgabe:
Spotlight-presentations:	1. 2.
Finale presentation / Abschlusspräsentation
Zweitprüfer / Second Examiner
Confidential / Vertraulich	No

Zeitplanung:

Checklist

Introduction / tour in M4
Urheberrechtsvereinbarung signed: https://www.tuhh.de/t3resources/sls/pdf/ZPA/Formulare_oeffentlich/Rechte_an_Abschlussarbeiten.pdf
if applicable: signed confidential agreement
official registration