Get 20M+ Full-Text Papers For Less Than $1.50/day. Start a 7-Day Trial for You or Your Team.

Learn More →

Physics-Informed Graph Learning for Shape Prediction in Robot Manipulate of Deformable Linear Objects

Physics-Informed Graph Learning for Shape Prediction in Robot Manipulate of Deformable Linear... Shape prediction of deformable linear objects (DLO) plays critical roles in robotics, medical devices, aerospace, and manufacturing, especially in manipulating objects such as cables, wires, and fibers. Due to the inherent flex - ibility of DLO and their complex deformation behaviors, such as bending and torsion, it is challenging to predict their dynamic characteristics accurately. Although the traditional physical modeling method can simulate the com- plex deformation behavior of DLO, the calculation cost is high and it is difficult to meet the demand of real-time prediction. In addition, the scarcity of data resources also limits the prediction accuracy of existing models. To solve these problems, a method of fiber shape prediction based on a physical information graph neural network (PIGNN) is proposed in this paper. This method cleverly combines the powerful expressive power of graph neural networks with the strict constraints of physical laws. Specifically, we learn the initial deformation model of the fiber through graph neural networks (GNN) to provide a good initial estimate for the model, which helps alleviate the problem of data resource scarcity. During the training process, we incorporate the physical prior knowledge of the dynamic deformation of the fiber optics into the loss function as a constraint, which is then fed back to the net - work model. This ensures that the shape of the fiber optics gradually approaches the true target shape, effectively solving the complex nonlinear behavior prediction problem of deformable linear objects. Experimental results dem- onstrate that, compared to traditional methods, the proposed method significantly reduces execution time and pre - diction error when handling the complex deformations of deformable fibers. This showcases its potential application value and superiority in fiber manipulation. Keywords Deformable linear objects, Fiber, Physics-informed graph neural network (PIGNN), Shape prediction 1 Introduction Deformable linear objects (DLO) refer to one-dimen- sional deformable entities, such as wires, ropes, and optical fibers. The shape prediction of DLO involves the predictive analysis of the deformation and motion of *Correspondence: deformable objects with linear properties under exter- Junliang Wang nal forces, which holds significant research value in [email protected] fields such as robotics, medical devices, aerospace, and Engineering Research Center of Artificial Intelligence for Textile Industry Ministry of Education, Institute of Artificial Intelligence, Donghua manufacturing. For instance, in wire manufacturing, University, Shanghai 201620, China shape prediction is utilized for the assembly of devices College of Mechanical Engineering, Donghua University, [1]; in surgical procedures, sutures are predicted in Shanghai 201620, China Central South Architectural Design Institute Co., Ltd., Wuhan 434400, order to secure tissue together [2]. However, the inher- China ent flexibility of DLO presents unique challenges for © The Author(s) 2025. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 2 of 12 shape prediction. Different from rigid objects, flex - the complex nonlinear deformation of fiber under exter - ible learning bodies may produce a variety of complex nal forces. The main contributions are as follows. deformations, such as bending and twisting. Accurate We developed a Physics-informed Graph Neural Net- prediction of the dynamic deformation of such objects work (PIGNN) model that integrates physical laws with requires the establishment of models that can predict graph neural networks, effectively enhancing the predic - complex deformation in real time, which makes this tive capability for the complex nonlinear deformation research field a key problem in robot control. behavior of fibers. By introducing physical constraints in In shape prediction for DLO, a major challenge is the training process, the dependence on a large number the nonlinear deformation such as bending and tor- of experimental data is reduced, the physical interpret- sion. When external conditions or constraints change, ability of the model is enhanced, and the dynamic defor- the DLO experiences complex nonlinear deformations. mation of DLO is captured more accurately. Because the traditional data-driven method [3–5] can- This study establishes a set of dynamics partial differ - not fully consider the deformation characteristics of ential equations that reflect the characteristics of fiber the DLO of materials such as optical fiber, it may make elastic objects, clearly describing the deformation and wrong prediction of the deformation of DLO, result- motion of fibers under external forces. These equations ing in control failure and even damage to the DLO. provide a theoretical foundation for the PIGNN, ensur- Although traditional physical modeling methods such ing that the model outputs align with actual physical phe- as finite element analysis [6 –9] can simulate these com- nomena, thus enhancing the accuracy and reliability of plex phenomena, their computational costs are high shape prediction, particularly in handling complex non- and it is difficult to meet the needs of real-time control. linear deformations. Therefore, how to efficiently capture the complex non - linear dynamics in fiber deformation has become an 2 Related Work important topic to improve the accuracy and efficiency Current methods for shape prediction of DLO can be of the DLO. primarily categorized into data-driven models and physi- Another challenge is the scarcity of data resources. cal models. These two approaches utilize different repre - Modeling of DLO requires a large amount of high-qual- sentations to simulate the complex dynamics of DLO. In ity experimental data for training and verification, and order to leverage the advantages of both physical models the acquisition of these data is usually costly and time- and data-driven models, we have developed a physics- consuming. Capturing multiple deformations of flexible informed neural network model. objects requires complex experimental equipment and accurate measurement techniques, which increases the 2.1 Data‑Driven Models difficulty of data acquisition. The scarcity of data directly In data-driven approaches, a common approach is the limits the accuracy of existing models, especially when method combining forward kinematics and model pre- dealing with complex deformation and nonlinear phe- diction (FKM+MPC). It acquires an accurate forward nomena. Therefore, how to construct an accurate model kinematics model through offline learning and makes under the condition of limited data has become another real-time adjustments using model predictive control important challenge of current research. during operation [11–13]. However, although this allows In recent years, the Physics-Informed Neural Network for the accurate learning of the forward kinematics of (PINN) [10] has emerged as a promising solution to vari- offline trained flexible bodies, problems may arise when ous challenges in modeling. PINNs incorporate physical operating different untrained flexible bodies due to the laws, often represented by partial differential equations lack of understanding of the physical properties of the (PDEs), directly into the architecture of neural networks. flexible bodies. Reinforcement learning methods have This integration allows the models to adhere to physi - also been studied [14, 15], but their data efficiency is rela - cal laws during training, thereby reducing the need for tively low, and it is challenging to transfer from trained extensive data sets. By leveraging the constraints imposed scenarios to untrained scenarios. by PDEs, PINNs can effectively learn the fundamental In addition to these offline methods, some studies use physical laws governing a system and accurately predict purely online methods to estimate the local linear defor- deformation shapes with minimal data. This approach is mation model of the manipulated flexible body, which particularly advantageous for capturing complex nonlin- can be applied to any new flexible body [16–18]. How - ear deformations. ever, these online estimation models have relatively low In this paper, we propose a fiber shape prediction accuracy because they can only utilize a small amount of method based on physical Infographic neural network local data. Some other studies use the method of offline (PIGNN), which aims to efficiently and accurately capture +Online Residual (GNN+Online Residual) to estimate W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 3 of 12 and reverse problems of the spatiotemporal fractional the local linear deformation model of the manipulated convection-diffusion equation by combining automatic flexible body [19]. Among them, graph neural networks differentiation and numerical discretization methods. (GNNS) have been proven to be capable of effectively Cai et  al. [24] conducted research on heat conduction, representing complex dynamic systems [20], and online which demonstrated the wide application of physical learning can also adjust the model in a timely manner to information neural networks in the engineering field. cope with new situations. However, in practical applica- In addition, Mao et  al. [25] applied PINN to solve the tions, the update speed of the model may not be sufficient complex dynamics of high-speed flows. These stud - to keep pace with the rate of dynamic changes in the sys- ies have shown the potential of physical information tem during practical applications. Therefore, adaptability neural networks in various scientific and engineering may be limited in rapidly changing environments. problems, laying a theoretical foundation for the devel- opment of this study. 2.2 Physical Models Shape prediction based on physical models enhances model interpretability, which could consist of finite 2.4 Analysis and Summary element methods and mass-spring-damper models. In summary, the current DLO shape prediction method In  Ref. [6], a finite element method (FEM) simulation still has the following shortcomings: of DLO is utilized for open-loop shape prediction. Fur- thermore, Ref.  [7] presents a method using simplified 1) The data-driven approach ignores the complex FEM for closed-loop shape prediction of DLO. Yoshida deformation characteristics of DLO. While forward et  al.  [8] established a model of circular DLO through kinematics models are effective at capturing trained finite element modeling, successfully demonstrating fiber deformation dynamics, a lack of understanding experiments in a simulated environment where an elas- of physical properties can lead to inaccurate predic- tic O-ring is placed around a cylinder. Frugoli et  al. [9] tions when faced with untrained fiber flexures. employed a discrete model to represent the behavior of 2) Physical models are deficient in prediction effi - simulated DLO. Due to the complexity and variability ciency. Although finite element methods provide of DLO deformations, modeling poses significant chal - interpretability and authenticity, their establishment lenges. Consequently, some researchers have proposed process usually requires a lot of computing resources, various methods to simulate the deformations of DLO. and it is difficult to achieve real-time control, which Among them, the mass spring damper model is mostly limits their application in fast response scenarios. used to simulate various linear objects [21, 22], such as cables and hair. In terms of medical treatment, medical robots as auxiliary devices can improve the precision The development of physical information neural network and speed of sutures, thus effectively improving the suc - (PINN) provides a new way to solve these problems. PINN cess rate of surgery. Both the mass spring damper model combines the advantages of deep learning with physi- and the finite element method have their advantages and cal constraints to effectively learn from small amounts of disadvantages. The mass spring damper model is more experimental data while maintaining adherence to physical intuitive and easier to implement than the finite element laws, improving adaptability in rapidly changing environ- model, which can produce more realistic physical simu- ments, combining the robustness of physical models with lation effects. However, building finite element models the flexibility of data-driven methods, and opening up new often requires a lot of computational resources, so apply- research methods for shape prediction of DLO. ing them to real-time robotic manipulation tasks is a Therefore, this paper proposes a shape prediction challenge. method for optical fibers based on Physics-Informed Graph Neural Networks (PIGNN). We first employ a Graph Neural Network (GNN) to capture the local and 2.3 Ph ysics‑Informed Neural Network Models global interactions of the optical fiber and utilize infor - In recent years, many studies have been conducted mation transfer between nodes to simulate the overall on physical-information neural networks (PINN) and dynamic behavior of the fiber, providing a good initial their variants. Raissi et  al. [10] proposed a frame- estimate. Then, during the training process, we incor - work for physical information neural networks that porate prior physical knowledge of the fiber’s dynamic solves both forward and backward problems by com- deformation as constraints into the loss function, feeding bining deep learning with nonlinear partial differen - this back to the network model to ensure that the pre- tial equations. Pang et  al. [23] extended the Physical dicted shape of the fiber gradually approaches the true Information Neural Networks (PINNs) to fractional target shape. PINNs(fPINNs), and successfully solved the forward Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 4 of 12 Coiling Reel End-Effector Connector Figure 2 Deformation of the optical fiber under end effector control Optical Fiber Clip If we denote the positions of the key points on the Figure 1 Robotic fiber coiling diagrams optical fiber as X =[x , x , ··· , x ] and the veloc- 1 2 n ity vectors of the key points on the optical fiber as X =[x˙ , x˙ , ··· , x˙ ] , the relationship between the posi- 1 2 n tion of the optical fiber at the current time and its posi - tion at the next time step can be expressed as: 3 Fiber Shape Prediction Model for the Robotic Coiling X(t + 1) = X(t) + X(t)�t. (1) 3.1 Robotic F iber Coiling Model From Figure 2, it is evident that there exists a mapping Fiber coiling refers to the process of winding optical relationship between the velocity of the robot’s end effec - fibers along a specific path to form a helical or circu - tor and the velocities of various points on the deform- lar shape, allowing for compact storage while prevent- able objects of the optical fiber. We assume the existence ing excessive bending that may result in attenuation of a function that describes the relationship between or signal loss. In this process, a robotic arm replaces the velocity vectors of the key points on the deformable traditional manual methods for intelligent fiber coil - objects of the optical fiber and the velocity vector of the ing, as illustrated in Figure  1. One end of the fiber is end effector, based on the known positions of the key fixed to the connector at the center of the coiling reel, points on the current deformable objects of the optical while the other end is securely held by the end effector fiber: of the robotic arm, with multiple clips ensuring the fib - er’s shape and position. During the coiling process, the ˙ ˙ X = f (X , R). (2) force applied by the robotic arm’s end effector dynami - cally adjusts according to variations in the path, sig- nificantly influencing the forces experienced at various 4 Fiber Shape Prediction Method Based points along the fiber. Therefore, to ensure the smooth on Physics‑Informed Graph Neural Networks progression of the coiling process, the model must (PIGNN) accurately calculate and predict the distribution of The Physics-informed Graph Neural Network (PIGNN) these forces and their effects on the fiber’s morphology. for fiber shape prediction is an innovative model that integrates physical constraints with graph neural net- works. In order to accurately describe the nonlinear 3.2 F iber Shape Prediction Model behavior of optical fibers under complex mechanical To investigate the impact of the forces applied by the conditions, a physical model reflecting the physical prop - robot’s end effector on the deformable objects of the erties of optical fibers is constructed in the partial differ - optical fiber, we set up the experimental environment ential equation (PDE). Then, the graph neural network on a flat tabletop without a coiling reel. In this envi - model (GNN) is used to learn and simulate the shape ronment, one end of the optical fiber is fixed, and the change of optical fiber, and the physical law is introduced other end is moved under the traction of the robot, and as the constraint, so that it can be effectively adjusted its shape change from time t to time t + 1 is observed. based on the physical law, so as to enhance the accu- Where,  X (t) represents the position of each key point racy and stability of the model and improve the predic- x (t) on the fiber at time t, X (t + 1) represents the posi- i tion ability of the network. By combining the physical tion of each key point x (t + 1) on the fiber at time i constraints with the learning ability of the graph neural t + 1 , and R(t) represents the velocity at the end of the network, PIGNN can accurately capture the global mor- robot at time t. The deformation of the optical fiber is phological changes of the fiber, ensuring the stability and illustrated in Figure 2. ... W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 5 of 12 accuracy of the model. The fiber shape prediction model state to X (t) , while incorporating the current X (t − m) based on physical information neural network estab- velocity information of the robot’s end effector. lished by us is shown in Figure  3, which is mainly com- ˙ ˙ X (t) = f (X (t − m : t), R(t)). (3) posed of three modules: Graph neural network (GNN), partial differential equation (PDE) of fiber flexible body The dynamic deformation of the optical fiber dynamics, and Loss function (Loss). under robot traction can be represented as a graph G = (V , E) , where the vertices V = (v , v , ··· , v ) cor- 1 2 n 4.1 Ph ysics‑Informed Graph Neural Networks respond to the key points and the robot control inputs, Studies show that graph neural networks (GNNS) are and the edges e = X (t) − X (t) correspond to the i,j i j applicable to the modeling of complex dynamic systems interactions among these key points. These edges rep - with structured and topological characteristics, such as resent the relative movements between the key points, flexible bodies and fluid dynamics, etc. [20]. In the con - with an edge being constructed when two key points text of fiber shape prediction, the physical behaviors of are within a “connection radius,” forming the topol- the optical fiber, such as bending and stretching, can be ogy of the graph. To learn and model this graph struc- conceptualized as a complex graph structure where the ture, we follow the encoding, processing, and decoding key points on the fiber are considered as nodes, and the architecture outlined in Ref. [20]. connections between the nodes represent the interac- Encoding Module The encoder module consists of a tions among these key points. The advantage of GNN vertex autoencoder and an edge autoencoder, which lies in their ability to capture both local and global inter- process the information of the graph’s vertices V and actions within the optical fiber, simulating the overall edges E , respectively. The output of the encoder is a dynamic behavior through information propagation low-dimensional representation of the graph, namely among the nodes. Therefore, in this paper, we adopt a 0 0 the encoded vertex features v and edge features e . i i,j GNN model similar to that in Ref. [20] as the neural net- These outputs will serve as inputs to the first informa - work component of the Physics-Informed Neural Net- tion propagation module in the processing module. work (PINN) to learn the dynamic deformation of the Information Propagation Module The dynamics of the optical fiber’s deformable objects. optical fiber are simulated by computing the interac - We assume that the relationship connecting the tions among the vertices in the latent graph. The out - robot’s end effector and the grasping point remains put after the kth message passing module consists of fixed throughout the execution of the task. The dynamic k+1 updated latent vertex features v and edge features function f (·) describes the interaction between the key k+1 e , which will be the input for the self-attention points on the optical fiber and the robot’s end effec - i,j mechanism. tor, extracting (m+1) states of the optical fiber from its GNN PDE Latent- features Auto matic K Times E D Message Differ Passing entiati on Target input:X,R update Random move Loss Data Loss Physical Loss Figure 3 Fiber shape prediction model based on physics-informed graph neural networks (PIGNN) Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 6 of 12  � k+1 v k k coating layer  v = f (v , e ),  i i i,j (4) k+1 e k k k cladding layer e = f (e , v , v ), i,j i j i,j v e core where f (·) and f (·) denote the features of the current k k vertex and edge, respectively, v and e denote the latent i i,j vertex and edge features after the kth message passing. The term e denotes the sum of the features of the i,j k k edges connected to vertex i, v and v are the features of i j Figure 5 Schematic diagram of optical fiber structure the two vertices i and j connected by edge (i, j). Decoding Module The decoder module reconstructs the obtained latent representation back into predictions of the original optical fiber motion, with the output being and nylon, designed to protect the optical fiber from the velocity information X (t) for each key point on the mechanical damage. This paper does not address the optical fiber at a specific time step. influence of the cladding on the refractive index of the fiber core but focuses mainly on the material character - 4.2 D ynamics Partial Differential Equations of Optical istics of the coating layer. Fiber Deformable Objects We used a UTM-5000 universal testing machine (accu- During the actual process of robotically pulling the opti- racy ±0.5%) to perform tensile tests on 20 samples of cal fiber, it becomes evident that the dynamic deforma - 0.6  m long fibers at a constant strain rate of 5  mm/min. tion of the fiber is highly complex. Predicting its shape Each optical fiber specimen is installed on the testing is crucial for successful operation, and the dynamic device with a pre-tension of 1 N to eliminate relaxation. behavior of the optical fiber can be described by stand - After 20 repeated tests, the elastic modulus was meas- ard dynamic partial differential equations. As a one- ured to be 2.5 GPa. The density was measured to be dimensional deformable object, the motion of the 1.2 g/cm using a density analyzer. optical fiber adheres to the continuum mechanic model. The material of the coating layer can be modeled as a In this context, the deformation of the fiber is smooth hyperelastic material, which exhibits highly nonlinear and predictable. characteristics in elastic deformation. For simplifica - In this paper, the mechanical model of the single- tion, this study adopts the Neo-Hookean model to simu- mode optical fiber (as shown in Figure  4) was simulated late the deformation of a 0.6  m long optical fiber with a by using the Abaqus software. The optical fiber is com - diameter of 1 mm. In the Abaqus software, the material posed of a core, a cladding layer and a coating layer, properties of the fiber are set as follows: elastic modu - and is a concentric cylinder, as shown in Figure  5. The lus E = 2.5 GPa , Poisson’s ratio ν = 0.38 , and density commonly used quartz optical fiber’s core and clad - ρ = 1.2 g/cm . The strain energy density function is ding are made of high-purity quartz glass with a small given by: amount of dopants. The dopants are used to ensure that the refractive index of the core is slightly higher � = C (I − 3) + (J − 1) , 10 1 (5) than that of the cladding. The coating layer is primarily composed of a specific ratio of acrylate, silicone resin, where, I = tr(C) is the first strain invariant (where C is the right Cauchy-Green tensor), and J = det(F ) is the determinant of the deformation gradient tensor F . The material parameters are determined by the elastic modu- lus E and Poisson’s ratio ν: E 6(1 − 2ν) C = , D = . 10 1 (6) 4(1 + ν) E In the simulation, the fiber is discretized into a struc - tured mesh of 200x20 (C3D8R elements). The bound - ary conditions are set as follows: all degrees of freedom are constrained on the left end, while a transverse con- Figure 4 Single-mode optical fiber diagram centrated force F = 1N is applied to the right end. The W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 7 of 12 the fiber, and A is the cross-sectional area. f denotes ext the external load, and N is the internal force vector. Among them, f can be represented by the speed ext R(t) at which the robot pulls the optical fiber at time t and the damping matrix D of the environmental model: f = DR(t). (8) ext And N includes the contributions of axial force and bending moment: N = EA ∈−EIκ , (9) where, E is the Young’s modulus, I is the moment of iner- ∂u(x,t) tia of the cross-section, the axial strain ∈= , and the ∂x ∂ u(x,t) curvature κ ≈ . ∂x Figure 6 Fiber optic displacement cloud map u Th s, when a force is applied at the end of the fiber, the dynamic partial differential equation governing the deformation of the fiber can be expressed as: 2 2 4 ∂ u(x, t) ∂ u(x, t) ∂ u(x, t) ρA = EA − EI + DR(t), 2 2 4 ∂t ∂x ∂x (10) ∂ u(x,t) where, EA represents the axial force of the fiber, ∂x consistent with the fundamental laws of elasticity, while ∂ u(x,t) −EI describes the fiber’s resistance to bending, ∂x effectively counteracting changes in curvature. 4.3 Loss Function for Physics‑Informed Fusion In the context of information-physics fusion, the con- struction of the loss function is a central concept of the PIGNN methodology. The loss function in PIGNN typi - Figure 7 Fiber stress cloud map cally consists of two components: data fitting loss and physical constraint loss, ensuring that the neural net- resulting displacement and stress contours are shown in work can learn from data while also adhering to physical Figures 6 and 7, respectively. laws. For the dynamic modeling of the deformable optical The displacement cloud image and stress cloud image fiber system, we can incorporate the partial differential obtained by simulation intuitively show the deformation equations governing the optical fiber into the physical distribution and stress concentration area of the fiber constraint component of the PIGNN, enabling shape under the action of load, which provides a basis for veri- prediction and learning of complex dynamic behaviors fying the accuracy of the hyperelastic material model. through the neural network. Based on these results, we combine simulation observa- Data fitting losses are used to ensure that PIGNN can tion with theoretical modeling to achieve quantitative fit real observational data. In this study, the data are analysis and prediction of the dynamic behavior of opti- derived from the optical fiber displacement measured by cal fibers. We further adopt the Euler-Bernoulli beam the sensor. Data fitting loss can be defined as: theory to establish the dynamic control equation of the optical fiber: 1 Loss = xˆ (t) − x (t) , (11) d i i i=1 ∂ u(x, t) ∂N ρA = + f . (7) ext ∂t ∂x where x ˆ (t) is the position of a key point on the fiber predicted by the model, x (t) is the actual observed data where, u(x, t) represents the displacement vector of the of the point, and N is the number of key points on the fiber at spatial coordinate x and time t; ρ is the density of fiber. We approach this loss by minimizing it to approxi - mate the experimental data. Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 8 of 12 In order to integrate the fiber deformation dynam - ˙ ˙ X(t) = f (X(t − m : t); R(t); θ). (16) PIGNN ics information into PIGNN, physical constraint loss is needed to ensure that the prediction results of graph neural network satisfy the partial differential equation 5 Experiments of fiber deformation dynamics. Using this equation as a 5.1 Experimental Setup constraint, the physical loss can be defined as: N 2 2 2 4 1 ∂ uˆ (x , t ) ∂ uˆ (x , t ) ∂ uˆ (x , t ) j j j j j j (12) Loss = ρA − EA − EI − DR(t) , 2 2 4 N  ∂t ∂x ∂x j=1 where u ˆ (x , t ) is the position vector of the optical fiber We conducted both simulation and real-world experi- j j predicted by the neural network, and N is the number of ments to validate the proposed method. Although we sampled points at different times and locations, through established a finite element model of the optical fiber, which the residuals of the system of partial differential we only used the derived partial differential equations equations are computed. as physical constraints for the PIGNN. Therefore, in the The total loss function is a weighted sum of the data bullet simulation environment based on Obi [26], we fitting loss and the physical constraint loss: modeled the optical fiber with its ends grasped by two robotic arm end-effectors. We tested a 3D task, where Loss = αLoss + βLoss . d p (13) the environmental dimension was 3 and the control input dimension n was 12. In the actual experiment, the opti- To dynamically balance the contributions of data cal fiber was placed on a table, with one end grasped by a fitting loss Loss and physical constraint loss Loss , d p UR5 robotic arm and the other end fixed. Consequently, weighting parameters α and β are defined as normal - the environmental dimension was 2, and the control ized ratios of loss gradient norms: input dimension n was 3. The shape of the optical fiber  � � � � ∇ Loss was represented by 13 key points, whose positions were  θ p � �  α = , � � obtained by measuring markers on the fiber using a cali - � � ∇ Loss + ∇ Loss +∈ θ d θ p (14) brated KinectV2 camera. In the experiments, both data  � � ∇ Loss θ d  � � β = , acquisition frequency and control frequency were set at � � �∇ Loss � + ∇ Loss +∈ θ θ p 10 Hz. The performance of the PIGNN model is evaluated where,  ∇ Loss and ∇ Loss represent the gradients θ d θ p based on the number of successful task completions, the of data loss and physical loss with respect to network average error in shape prediction, and the average time parameters θ respectively, and ϵ is a minimal value (set taken for predictions. The definition of the average error −8 to ϵ=10 ). When the magnitude of physical constraint in shape prediction is as follows: gradients significantly exceeds that of data fitting gradi - 13 13 ents, α is reduced to prevent physical constraints from 2 2 2 ˆ ˆ E = |d − xˆ | = (x − d ) + (x − d ) , dominating the optimization process excessively, and vice p i i i_x i_x i_y i_y i=1 i=1 versa. Through backpropagation of physical constraints, (17) we optimize the network model by simultaneously refin - where E represents the sum of squared errors for all ing data gradients and physical consistency via gradient key points, x ˆ denotes the predicted position of each key descent. During backpropagation, the update direction of point, and d indicates the target position of each key model parameters θ is determined by the weighted sum point. of composite gradients: The training data was collected in a simulated environ - θ ← θ − η(α∇ Loss + β∇ Loss ), θ θ p d (15) ment where the clampers randomly moved the ends of the optical fiber. First, we trained a graph neural network where η denotes the learning rate. The physical informa - to learn the initial deformation model. Then, we adjusted tion graph neural network f (·) leverages Tensor- PIGNN the computed preliminary deformation using the infor- Flow’s automatic differentiation to compute ∇ Loss . θ p mation from physical constraints. We initially tested the After updating θ through backpropagation, the model’s accuracy of both the data model and the physical mode- final corrected prediction is expressed as: ling on a specific optical fiber. Subsequently, we collected 10000 data points from the same optical fiber, with 70% W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 9 of 12 of the data designated as the training dataset, 15% as the Table 1 Performance of methods in optical fiber shape prediction task in simulation testing dataset, and the remaining 15% as the validation dataset. For testing and validation, we used the trained Methods Success rate Average task Average model to predict the shape of the optical fiber after 10 error (cm) task time (s) steps. GNN+Online Residual 76/100 0.553 10.324 5.2 Results Online learning 56/100 5.681 35.895 FKM+MPC 65/100 2.426 21.088 1) Simulation First, we tested the performance of Pure physical 96/100 0.239 248.451 the PIGNN model in the 3D shape prediction task Ours(w/o physical) 58/100 2.833 18.256 within the simulation, as shown in Figure 8. The con - Ours 90/100 0.251 7.562 trolled optical fiber was untrained, and we tested five cases with different target shapes. We also per - formed ablation studies, as detailed in Table  1. Our ber of successes and prediction accuracy, but it method significantly outperformed the compara - takes too long. The offline + online residual method tive methods in terms of success rates and average (GNN + Online Residual) performed relatively well task errors, even with less offline training data. The but achieved fewer successes than our approach. online learning method had the highest task time Our model without physical information (Ours(w/o since it required initialization of the Jacobian matrix physical)) achieved fewer successful trials and had by moving the ends of the deformable objects along a higher average task error. In contrast, the method each degree of freedom at each start. The combina - with physical constraints reached a 90% success rate tion of forward kinematics and model predictive con- and an average task error of 0.251 cm, second only to trol methods (FKM + MPC) exhibited higher aver- the pure physical model method. age task errors because it did not further update for 2) Real-world Experiments We evaluated these meth- the untrained deformable objects. The Pure physical ods in real 2D tasks. Using the same model as in the model method performs best in terms of the num- Figure 8: Simulation of optical fiber shape prediction, a, b, c, d, e represent the five target shape prediction tasks we have established Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 10 of 12 Figure 9 Shape prediction task testing 3, as indicated in Table  2, results show that the pure online learning method progresses slowly and une- venly, while the approach combining forward kin- ematics and model predictive control (FKM+MPC) is fast but less accurate. The offline + online residual method (GNN + Online Residual) achieves similar success rates and average task times as our model. The average task error of the pure physical model method is the lowest, but the time consumption is also the longest. After removing the physical infor- mation (Ours(w/o physical)), our model still per- Figure 10 Case 1 (0.6 m fiber optic) formed poorly in the actual experiments, while our model with physical constraints was superior to the offline + online residual method in the average task error. The actual experimental results prove that our method has the highest number of successes among other methods except for the pure physical model and achieves the lowest average task error. The physi - cal constraints of optical fibers make the prediction faster and more accurate. Although the average task error of the pure physical model is the lowest and has high accuracy, its prediction speed is the slowest and does not meet the requirements of real-time control of optical fibers. Figure 11 Case 4 (1 m fiber optic) The main advantages of our method are reflected in several aspects: First, PIGNN effectively captures the complex nonlinear characteristics of the optical fiber simulation, we did not collect real-world data for during the winding process by incorporating physical training. Considering that one end of the optical fiber laws as constraints into the model, thereby enhancing is fixed during the winding process, we employed a the model’s robustness and reliability. Second, PIGNN single arm to perform operations and conducted five utilizes graph neural networks to extract global dynamic different feasible winding shape tests on two types of information, performing better than traditional data- optical fibers: one with a length of 0.6 m and a diam - driven methods under conditions of limited data. The eter of 1 mm, and another with a length of 1 m and a experimental results show that it is second only to the diameter of 1  mm, as shown in Figure  9. Figures  10 Pure physical model method in terms of task success rate and 11 visualize the prediction processes for both and average task error. Moreover, PIGNN introduces cases. Due to the control input dimension being only W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 11 of 12 Table 2 Performance in real-world 2D shape prediction tasks Methods case1(0.6 m fiber optic) case4 (1 m fiber optic) Success rate Average task Average task Success rate Average task Average error (cm) time (s) error (cm) task time (s) GNN+Online Residual 9/10 1.211 10.114 9/10 1.782 14.324 Online learning 8/10 7.442 22.895 7/10 6.311 25.432 FKM+MPC 8/10 2.997 12.047 8/10 3.626 16.758 Pure physical 10/10 0.985 384.532 10/10 1.246 667.289 Ours(w/o physical) 6/10 2.760 15.776 6/10 3.818 16.442 Ours 9/10 1.188 10.885 10/10 1.551 13.572 physical constraints as the loss function. In tasks involv- optical fibers and their surroundings, aiming to pro - ing complex deformation of optical fibers (such as case4), vide smarter solutions for optical fiber manipulation in the predicted deformation error is small, and it has the practical industrial scenarios. fastest prediction speed among all methods. In summary, PIGNN’s comprehensive performance in multiple tasks 1) Multi-step Operations (Grasp and Release Required) is superior to other methods, which proves its effective - Real-world optical fiber manipulation tasks may ness and superiority as an optical fiber shape prediction require multi-step shaping, necessitating changes model, and provides an efficient and accurate solution for in the grasping points during operation. Identifying intelligent disk fiber tasks. optimal grasping points for the optical fiber presents a challenging research problem. We plan to utilize Transporter Networks [27] to learn grasping points 6 Conclusions from images and integrate them with our manipula- In this paper, an innovative method based on physical tion approach. information graph neural network (PIGNN) is proposed 2) Optical Fiber Winding Environments Optical fiber to predict the shape of deformable linear objects, such as manipulation tasks predominantly involve winding optical fibers. Compared with the traditional data-driven processes and must consider complex environments. method, this method combines the powerful expression Model-based approaches are rarely proposed in the ability of graph neural network with the constraint of literature. Mitrano  et al. [28] introduced a method physical law, and provides a new perspective and solu- to train classifiers for determining effective positions tion for the deformation problem of deformable linear for offline learning models and employed learning objects. The graph neural network (GNN) is introduced strategies to recover robot functionality when the to learn the initial deformation model of the fiber. During model was unreliable. However, this method does the training process, the physical prior knowledge of the not adequately address scenarios where the optical dynamic deformation of the fiber is incorporated into the fiber needs to interact with the environment. We will loss function as a constraint and fed back to the network pursue this as a direction for future research. model, thus ensuring that the shape of the fiber is gradu - ally close to the real target shape. Acknowledgements The experimental results show that under the condi - Not applicable. tion of data scarcity, our method performs well in both prediction accuracy and stability, especially in the task Author contributions MW and JW determined the overall framework and central idea of the article; of complex deformation such as bending and twisting of MW was responsible for all experiments and analysis, and wrote and checked optical fiber. In comparison to conventional methods, the the manuscript; JZ and JW assisted with the manuscript; XL and GL were PIGNN model shows marked improvements in execution responsible for a part of experiments. All authors read and approved the final manuscript. time and prediction error, with strong performance in real-time predictions suggesting promising applications. Funding Future research will further extend the application of Supported by the Fundamental Research Funds for the Central Universities (Grant Nos. 2232024Y-01, LZB2023001), DHU Distinguished Young Professor this method in more complex three-dimensional envi- Program, National Natural Science Foundation of China (Grant No. 52275478), ronments and explore its adaptability and robustness AI-Enhanced Research Program of Shanghai Municipal Education Commission in dealing with the interactions between deformable (Grant No. SMEC-AI-DHUY-05) Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 12 of 12 Availability of data and materials [18] J Zhu, D Navarro-Alarcon, R Passama, et al. Vision-based manipulation of Not applicable. deformable and rigid objects using subspace projections of 2D contours. Robotics and Autonomous Systems, 2021, 142: 103798. [19] R Y Ming, H Z zhong, X Li. Shape control of deformable linear objects Declarations with offline and online learning of local linear deformation models. IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, Competing Interests PA, USA, May 23-27, 2022: 1337-1343. The authors declare no competing financial interests. [20] Y Z Li, J J Wu, R Tedrake, et al. Learning particle dynamics for manipulating th rigid bodies, deformable objects, and fluids. 7 International Conference on Learning Representations (ICLR), New Orleans, LA, USA, May 06-09, 2019. Received: 24 December 2024 Revised: 5 June 2025 Accepted: 11 June [21] H Wakamatsu, K Takahashi, S Hirai. Dynamic modeling of linear object deformation based on differential geometry coordinates. IEEE Interna- tional Conference on Robotics and Automation, Barcelona, Spain, April 18-22, 2005: 1028-1033. [22] F F Khalil, P Payeur. Dexterous robotic manipulation of deformable objects with multi-sensory feedback - a review. Robot Manipulators Trends References and Development. United States: INTECH Open Access Publisher, 2010: [1] X Li, X Su, Y Gao, et al. Vision-based robotic grasping and manipulation 587-621. of USB wires. IEEE International Conference on Robotics and Automation [23] G F Pang, L Lu, G E Karniadakis, et al. fPINNs: Fractional physics-informed (ICRA). Brisbane, QLD, Australia, May 21-25, 2018: 3482-3487. neural networks. SIAM Journal on Scientific Computing, 2019, 41(4): [2] L Cao, X Li, P T Phan, et al. Sewing up the wounds: A robotic suturing A2603-A2626. system for flexible endoscopy. IEEE Robotics & Automation Magazine, 2020, [24] S Z Cai, Z C Wang, S F Wang, et al. Physics-informed neural networks 27(3): 45-54. for heat transfer problems. Journal of Heat Transfer, 2021, 143(6): [3] R Lagneau, A Krupa, M Marchal. Automatic shape control of deformable 060801-060815. wires based on model-free visual servoing. IEEE Robotics and Automation [25] Z P Mao, A D Jagtap, G E Karniadakis. Physics-informed neural networks Letters, 2020, 5(4): 5252 - 5259. for high-speed flows. Computer Methods in Applied Mechanics and Engi- [4] Z Hu, T Han, P G Sun, et al. 3-D deformable object manipulation using deep neering, 2020, 360: 112789. neural networks. IEEE Robotics and Automation Letters, 2019, 4(4): 4255-4261. [26] V M Studio. Obi: Unified particle physics for Unity 3D, 2019. http:// obi. [5] C Morley, R V Patal. Steering of flexible needles using an LSTM encoder with virtu almet hodst udio. com. model predictive control. 2022 2nd International Conference on Robotics, [27] D Seita, P Florence, J Tompson, et al. Learning to rearrange deformable Automation and Artificial Intelligence (RAAI), Singapore, December 09-11, cables, fabrics, and bags with goal-conditioned transporter networks. 2022, 99-104. IEEE International Conference on Robotics and Automation (ICRA), Xi’an, [6] A Nealen, M Müller, R Keiser, et al. Physically based deformable models in China, May 30-June 05, 2021: 4568-4575. computer graphics. Computer Graphics Forum, 2006, 25(4): 809-836. [28] P Mitrano, D McConachie, D Berenson. Learning where to trust unreliable [7] F S Sin, D Schroeder, J Barbic. Vega: Non-linear FEM deformable object models in an unstructured world for deformable object manipulation. simulator. Computer Graphics Forum, 2013, 32(1): 36-48. Science Robotics, 2021, 6(54): eabd8170. [8] E Yoshida, K Ayusawa, I G Ramirez-Alpizar, et al. Simulation-based optimal motion planning for deformable object. IEEE International Workshop on Advanced Robotics and its Social Impacts (ARSO). Lyon, France, June 30-July Meixuan Wang born in 1999, is currently a master’s candidate at 02, 2015: 1-6. the College of Mechanical Engineering, Donghua University, China. Her [9] G Frugoli, A Galimberti, C Rizzi, et al. Discrete element approach for research focuses on machine vision in intelligent manufacturing non-rigid material modeling. Robot Manipulation of Deformable Objects. systems. Advanced Manufacturing, London: Springer, 2000: 29-41. [10] M Raissi, P Perdikaris, G E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems Junliang Wang born in 1991, is currently an associate researcher involving nonlinear partial differential equations. Journal of Computa- at Institute of Artificial Intelligence, Donghua University, China. He tional Physics, 2019, 378: 686-707. received his PhD from Shanghai Jiao Tong University, China. His [11] W Yan, A Vangipuram, P Abbeel, et al. Learning predictive representations research focuses on smart manufacturing and robotics, computer for deformable objects using contrastive estimation. Proceedings of the vision, and big data intelligence. Conference on Robot Learning. 2020, 155: 564-574. [12] W B Zhang, K Schmeckpeper, P Chaudhari, et al. Deformable linear object Jie Zhang born in 1963, is currently a professor and the Executive prediction using locally linear latent dynamics. 2021 IEEE International Dean at Institute of Artificial Intelligence, Donghua University, China. She Conference on Robotics and Automation (ICRA), Xi’an, China, May 30-June received her PhD from Nanjing University of Aeronautics and Astronau- 05, 2021: 13503-13509. [13] Y Yang, J A Stork, T Stoyanov. Learning to propagate interaction effects for tics, China. Her research interests include smart manufacturing and Modeling Deformable Linear Objects Dynamics. 2021 IEEE International robotics, big data intelligence, machine cognitive learning, and com- Conference on Robotics and Automation (ICRA), Xi’an, China, May 30-June plex system modeling and control. 05, 2021: 1950-1957. [14] A Ajay, J Wu, N Fazeli, et al. Augmenting physical simulators with stochas- Xinting Liao born in 1997, is currently a PhD candidate at College tic neural networks: Case study of planar pushing and bouncing. IEEE/RSJ of Mechanical Engineering, Donghua University, China. She received International Conference on Intelligent Robots and Systems (IROS), Madrid, her Master’s degree from South China Agricultural University, China. Spain, October 01-05, 2018: 3066-3073. Her research focuses on pattern recognition and quality control in [15] A Kloss, S Schaal, J Bohg. Combining learned and analytical models for intelligent manufacturing systems. predicting action effects from sensory data. The International Journal of Robotics Research, 2022, 41 (8): 778-797. [16] D Navarro-Alarcon, Y H Liu, J G Romero, et al. Model-free visually servoed Guojin Li born in 1984, is currently an engineer at the Central deformation control of elastic objects by robot manipulators. IEEE Trans. South Architectural Design Institute Co., Ltd., China. His research inter- Robotics, 2013, 29 (6): 1457-1468. ests include deep learning, computer vision, and natural language [17] J Zhu, B Navarro, P Fraisse, et al. Dual-arm robotic manipulation of flexible processing. cables. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, October 01-05, 2018, 479-484. http://www.deepdyve.com/assets/images/DeepDyve-Logo-lg.png Chinese Journal of Mechanical Engineering Springer Journals

Physics-Informed Graph Learning for Shape Prediction in Robot Manipulate of Deformable Linear Objects

Download PDF
12 pages

Loading next page...
 
/lp/springer-journals/physics-informed-graph-learning-for-shape-prediction-in-robot-VDJdlpoceh

References (25)

Publisher
Springer Journals
Copyright
Copyright © The Author(s) 2025
ISSN
1000-9345
eISSN
2192-8258
DOI
10.1186/s10033-025-01299-3
Publisher site
See Article on Publisher Site

Abstract

Shape prediction of deformable linear objects (DLO) plays critical roles in robotics, medical devices, aerospace, and manufacturing, especially in manipulating objects such as cables, wires, and fibers. Due to the inherent flex - ibility of DLO and their complex deformation behaviors, such as bending and torsion, it is challenging to predict their dynamic characteristics accurately. Although the traditional physical modeling method can simulate the com- plex deformation behavior of DLO, the calculation cost is high and it is difficult to meet the demand of real-time prediction. In addition, the scarcity of data resources also limits the prediction accuracy of existing models. To solve these problems, a method of fiber shape prediction based on a physical information graph neural network (PIGNN) is proposed in this paper. This method cleverly combines the powerful expressive power of graph neural networks with the strict constraints of physical laws. Specifically, we learn the initial deformation model of the fiber through graph neural networks (GNN) to provide a good initial estimate for the model, which helps alleviate the problem of data resource scarcity. During the training process, we incorporate the physical prior knowledge of the dynamic deformation of the fiber optics into the loss function as a constraint, which is then fed back to the net - work model. This ensures that the shape of the fiber optics gradually approaches the true target shape, effectively solving the complex nonlinear behavior prediction problem of deformable linear objects. Experimental results dem- onstrate that, compared to traditional methods, the proposed method significantly reduces execution time and pre - diction error when handling the complex deformations of deformable fibers. This showcases its potential application value and superiority in fiber manipulation. Keywords Deformable linear objects, Fiber, Physics-informed graph neural network (PIGNN), Shape prediction 1 Introduction Deformable linear objects (DLO) refer to one-dimen- sional deformable entities, such as wires, ropes, and optical fibers. The shape prediction of DLO involves the predictive analysis of the deformation and motion of *Correspondence: deformable objects with linear properties under exter- Junliang Wang nal forces, which holds significant research value in [email protected] fields such as robotics, medical devices, aerospace, and Engineering Research Center of Artificial Intelligence for Textile Industry Ministry of Education, Institute of Artificial Intelligence, Donghua manufacturing. For instance, in wire manufacturing, University, Shanghai 201620, China shape prediction is utilized for the assembly of devices College of Mechanical Engineering, Donghua University, [1]; in surgical procedures, sutures are predicted in Shanghai 201620, China Central South Architectural Design Institute Co., Ltd., Wuhan 434400, order to secure tissue together [2]. However, the inher- China ent flexibility of DLO presents unique challenges for © The Author(s) 2025. Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http:// creat iveco mmons. org/ licen ses/ by/4. 0/. Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 2 of 12 shape prediction. Different from rigid objects, flex - the complex nonlinear deformation of fiber under exter - ible learning bodies may produce a variety of complex nal forces. The main contributions are as follows. deformations, such as bending and twisting. Accurate We developed a Physics-informed Graph Neural Net- prediction of the dynamic deformation of such objects work (PIGNN) model that integrates physical laws with requires the establishment of models that can predict graph neural networks, effectively enhancing the predic - complex deformation in real time, which makes this tive capability for the complex nonlinear deformation research field a key problem in robot control. behavior of fibers. By introducing physical constraints in In shape prediction for DLO, a major challenge is the training process, the dependence on a large number the nonlinear deformation such as bending and tor- of experimental data is reduced, the physical interpret- sion. When external conditions or constraints change, ability of the model is enhanced, and the dynamic defor- the DLO experiences complex nonlinear deformations. mation of DLO is captured more accurately. Because the traditional data-driven method [3–5] can- This study establishes a set of dynamics partial differ - not fully consider the deformation characteristics of ential equations that reflect the characteristics of fiber the DLO of materials such as optical fiber, it may make elastic objects, clearly describing the deformation and wrong prediction of the deformation of DLO, result- motion of fibers under external forces. These equations ing in control failure and even damage to the DLO. provide a theoretical foundation for the PIGNN, ensur- Although traditional physical modeling methods such ing that the model outputs align with actual physical phe- as finite element analysis [6 –9] can simulate these com- nomena, thus enhancing the accuracy and reliability of plex phenomena, their computational costs are high shape prediction, particularly in handling complex non- and it is difficult to meet the needs of real-time control. linear deformations. Therefore, how to efficiently capture the complex non - linear dynamics in fiber deformation has become an 2 Related Work important topic to improve the accuracy and efficiency Current methods for shape prediction of DLO can be of the DLO. primarily categorized into data-driven models and physi- Another challenge is the scarcity of data resources. cal models. These two approaches utilize different repre - Modeling of DLO requires a large amount of high-qual- sentations to simulate the complex dynamics of DLO. In ity experimental data for training and verification, and order to leverage the advantages of both physical models the acquisition of these data is usually costly and time- and data-driven models, we have developed a physics- consuming. Capturing multiple deformations of flexible informed neural network model. objects requires complex experimental equipment and accurate measurement techniques, which increases the 2.1 Data‑Driven Models difficulty of data acquisition. The scarcity of data directly In data-driven approaches, a common approach is the limits the accuracy of existing models, especially when method combining forward kinematics and model pre- dealing with complex deformation and nonlinear phe- diction (FKM+MPC). It acquires an accurate forward nomena. Therefore, how to construct an accurate model kinematics model through offline learning and makes under the condition of limited data has become another real-time adjustments using model predictive control important challenge of current research. during operation [11–13]. However, although this allows In recent years, the Physics-Informed Neural Network for the accurate learning of the forward kinematics of (PINN) [10] has emerged as a promising solution to vari- offline trained flexible bodies, problems may arise when ous challenges in modeling. PINNs incorporate physical operating different untrained flexible bodies due to the laws, often represented by partial differential equations lack of understanding of the physical properties of the (PDEs), directly into the architecture of neural networks. flexible bodies. Reinforcement learning methods have This integration allows the models to adhere to physi - also been studied [14, 15], but their data efficiency is rela - cal laws during training, thereby reducing the need for tively low, and it is challenging to transfer from trained extensive data sets. By leveraging the constraints imposed scenarios to untrained scenarios. by PDEs, PINNs can effectively learn the fundamental In addition to these offline methods, some studies use physical laws governing a system and accurately predict purely online methods to estimate the local linear defor- deformation shapes with minimal data. This approach is mation model of the manipulated flexible body, which particularly advantageous for capturing complex nonlin- can be applied to any new flexible body [16–18]. How - ear deformations. ever, these online estimation models have relatively low In this paper, we propose a fiber shape prediction accuracy because they can only utilize a small amount of method based on physical Infographic neural network local data. Some other studies use the method of offline (PIGNN), which aims to efficiently and accurately capture +Online Residual (GNN+Online Residual) to estimate W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 3 of 12 and reverse problems of the spatiotemporal fractional the local linear deformation model of the manipulated convection-diffusion equation by combining automatic flexible body [19]. Among them, graph neural networks differentiation and numerical discretization methods. (GNNS) have been proven to be capable of effectively Cai et  al. [24] conducted research on heat conduction, representing complex dynamic systems [20], and online which demonstrated the wide application of physical learning can also adjust the model in a timely manner to information neural networks in the engineering field. cope with new situations. However, in practical applica- In addition, Mao et  al. [25] applied PINN to solve the tions, the update speed of the model may not be sufficient complex dynamics of high-speed flows. These stud - to keep pace with the rate of dynamic changes in the sys- ies have shown the potential of physical information tem during practical applications. Therefore, adaptability neural networks in various scientific and engineering may be limited in rapidly changing environments. problems, laying a theoretical foundation for the devel- opment of this study. 2.2 Physical Models Shape prediction based on physical models enhances model interpretability, which could consist of finite 2.4 Analysis and Summary element methods and mass-spring-damper models. In summary, the current DLO shape prediction method In  Ref. [6], a finite element method (FEM) simulation still has the following shortcomings: of DLO is utilized for open-loop shape prediction. Fur- thermore, Ref.  [7] presents a method using simplified 1) The data-driven approach ignores the complex FEM for closed-loop shape prediction of DLO. Yoshida deformation characteristics of DLO. While forward et  al.  [8] established a model of circular DLO through kinematics models are effective at capturing trained finite element modeling, successfully demonstrating fiber deformation dynamics, a lack of understanding experiments in a simulated environment where an elas- of physical properties can lead to inaccurate predic- tic O-ring is placed around a cylinder. Frugoli et  al. [9] tions when faced with untrained fiber flexures. employed a discrete model to represent the behavior of 2) Physical models are deficient in prediction effi - simulated DLO. Due to the complexity and variability ciency. Although finite element methods provide of DLO deformations, modeling poses significant chal - interpretability and authenticity, their establishment lenges. Consequently, some researchers have proposed process usually requires a lot of computing resources, various methods to simulate the deformations of DLO. and it is difficult to achieve real-time control, which Among them, the mass spring damper model is mostly limits their application in fast response scenarios. used to simulate various linear objects [21, 22], such as cables and hair. In terms of medical treatment, medical robots as auxiliary devices can improve the precision The development of physical information neural network and speed of sutures, thus effectively improving the suc - (PINN) provides a new way to solve these problems. PINN cess rate of surgery. Both the mass spring damper model combines the advantages of deep learning with physi- and the finite element method have their advantages and cal constraints to effectively learn from small amounts of disadvantages. The mass spring damper model is more experimental data while maintaining adherence to physical intuitive and easier to implement than the finite element laws, improving adaptability in rapidly changing environ- model, which can produce more realistic physical simu- ments, combining the robustness of physical models with lation effects. However, building finite element models the flexibility of data-driven methods, and opening up new often requires a lot of computational resources, so apply- research methods for shape prediction of DLO. ing them to real-time robotic manipulation tasks is a Therefore, this paper proposes a shape prediction challenge. method for optical fibers based on Physics-Informed Graph Neural Networks (PIGNN). We first employ a Graph Neural Network (GNN) to capture the local and 2.3 Ph ysics‑Informed Neural Network Models global interactions of the optical fiber and utilize infor - In recent years, many studies have been conducted mation transfer between nodes to simulate the overall on physical-information neural networks (PINN) and dynamic behavior of the fiber, providing a good initial their variants. Raissi et  al. [10] proposed a frame- estimate. Then, during the training process, we incor - work for physical information neural networks that porate prior physical knowledge of the fiber’s dynamic solves both forward and backward problems by com- deformation as constraints into the loss function, feeding bining deep learning with nonlinear partial differen - this back to the network model to ensure that the pre- tial equations. Pang et  al. [23] extended the Physical dicted shape of the fiber gradually approaches the true Information Neural Networks (PINNs) to fractional target shape. PINNs(fPINNs), and successfully solved the forward Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 4 of 12 Coiling Reel End-Effector Connector Figure 2 Deformation of the optical fiber under end effector control Optical Fiber Clip If we denote the positions of the key points on the Figure 1 Robotic fiber coiling diagrams optical fiber as X =[x , x , ··· , x ] and the veloc- 1 2 n ity vectors of the key points on the optical fiber as X =[x˙ , x˙ , ··· , x˙ ] , the relationship between the posi- 1 2 n tion of the optical fiber at the current time and its posi - tion at the next time step can be expressed as: 3 Fiber Shape Prediction Model for the Robotic Coiling X(t + 1) = X(t) + X(t)�t. (1) 3.1 Robotic F iber Coiling Model From Figure 2, it is evident that there exists a mapping Fiber coiling refers to the process of winding optical relationship between the velocity of the robot’s end effec - fibers along a specific path to form a helical or circu - tor and the velocities of various points on the deform- lar shape, allowing for compact storage while prevent- able objects of the optical fiber. We assume the existence ing excessive bending that may result in attenuation of a function that describes the relationship between or signal loss. In this process, a robotic arm replaces the velocity vectors of the key points on the deformable traditional manual methods for intelligent fiber coil - objects of the optical fiber and the velocity vector of the ing, as illustrated in Figure  1. One end of the fiber is end effector, based on the known positions of the key fixed to the connector at the center of the coiling reel, points on the current deformable objects of the optical while the other end is securely held by the end effector fiber: of the robotic arm, with multiple clips ensuring the fib - er’s shape and position. During the coiling process, the ˙ ˙ X = f (X , R). (2) force applied by the robotic arm’s end effector dynami - cally adjusts according to variations in the path, sig- nificantly influencing the forces experienced at various 4 Fiber Shape Prediction Method Based points along the fiber. Therefore, to ensure the smooth on Physics‑Informed Graph Neural Networks progression of the coiling process, the model must (PIGNN) accurately calculate and predict the distribution of The Physics-informed Graph Neural Network (PIGNN) these forces and their effects on the fiber’s morphology. for fiber shape prediction is an innovative model that integrates physical constraints with graph neural net- works. In order to accurately describe the nonlinear 3.2 F iber Shape Prediction Model behavior of optical fibers under complex mechanical To investigate the impact of the forces applied by the conditions, a physical model reflecting the physical prop - robot’s end effector on the deformable objects of the erties of optical fibers is constructed in the partial differ - optical fiber, we set up the experimental environment ential equation (PDE). Then, the graph neural network on a flat tabletop without a coiling reel. In this envi - model (GNN) is used to learn and simulate the shape ronment, one end of the optical fiber is fixed, and the change of optical fiber, and the physical law is introduced other end is moved under the traction of the robot, and as the constraint, so that it can be effectively adjusted its shape change from time t to time t + 1 is observed. based on the physical law, so as to enhance the accu- Where,  X (t) represents the position of each key point racy and stability of the model and improve the predic- x (t) on the fiber at time t, X (t + 1) represents the posi- i tion ability of the network. By combining the physical tion of each key point x (t + 1) on the fiber at time i constraints with the learning ability of the graph neural t + 1 , and R(t) represents the velocity at the end of the network, PIGNN can accurately capture the global mor- robot at time t. The deformation of the optical fiber is phological changes of the fiber, ensuring the stability and illustrated in Figure 2. ... W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 5 of 12 accuracy of the model. The fiber shape prediction model state to X (t) , while incorporating the current X (t − m) based on physical information neural network estab- velocity information of the robot’s end effector. lished by us is shown in Figure  3, which is mainly com- ˙ ˙ X (t) = f (X (t − m : t), R(t)). (3) posed of three modules: Graph neural network (GNN), partial differential equation (PDE) of fiber flexible body The dynamic deformation of the optical fiber dynamics, and Loss function (Loss). under robot traction can be represented as a graph G = (V , E) , where the vertices V = (v , v , ··· , v ) cor- 1 2 n 4.1 Ph ysics‑Informed Graph Neural Networks respond to the key points and the robot control inputs, Studies show that graph neural networks (GNNS) are and the edges e = X (t) − X (t) correspond to the i,j i j applicable to the modeling of complex dynamic systems interactions among these key points. These edges rep - with structured and topological characteristics, such as resent the relative movements between the key points, flexible bodies and fluid dynamics, etc. [20]. In the con - with an edge being constructed when two key points text of fiber shape prediction, the physical behaviors of are within a “connection radius,” forming the topol- the optical fiber, such as bending and stretching, can be ogy of the graph. To learn and model this graph struc- conceptualized as a complex graph structure where the ture, we follow the encoding, processing, and decoding key points on the fiber are considered as nodes, and the architecture outlined in Ref. [20]. connections between the nodes represent the interac- Encoding Module The encoder module consists of a tions among these key points. The advantage of GNN vertex autoencoder and an edge autoencoder, which lies in their ability to capture both local and global inter- process the information of the graph’s vertices V and actions within the optical fiber, simulating the overall edges E , respectively. The output of the encoder is a dynamic behavior through information propagation low-dimensional representation of the graph, namely among the nodes. Therefore, in this paper, we adopt a 0 0 the encoded vertex features v and edge features e . i i,j GNN model similar to that in Ref. [20] as the neural net- These outputs will serve as inputs to the first informa - work component of the Physics-Informed Neural Net- tion propagation module in the processing module. work (PINN) to learn the dynamic deformation of the Information Propagation Module The dynamics of the optical fiber’s deformable objects. optical fiber are simulated by computing the interac - We assume that the relationship connecting the tions among the vertices in the latent graph. The out - robot’s end effector and the grasping point remains put after the kth message passing module consists of fixed throughout the execution of the task. The dynamic k+1 updated latent vertex features v and edge features function f (·) describes the interaction between the key k+1 e , which will be the input for the self-attention points on the optical fiber and the robot’s end effec - i,j mechanism. tor, extracting (m+1) states of the optical fiber from its GNN PDE Latent- features Auto matic K Times E D Message Differ Passing entiati on Target input:X,R update Random move Loss Data Loss Physical Loss Figure 3 Fiber shape prediction model based on physics-informed graph neural networks (PIGNN) Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 6 of 12  � k+1 v k k coating layer  v = f (v , e ),  i i i,j (4) k+1 e k k k cladding layer e = f (e , v , v ), i,j i j i,j v e core where f (·) and f (·) denote the features of the current k k vertex and edge, respectively, v and e denote the latent i i,j vertex and edge features after the kth message passing. The term e denotes the sum of the features of the i,j k k edges connected to vertex i, v and v are the features of i j Figure 5 Schematic diagram of optical fiber structure the two vertices i and j connected by edge (i, j). Decoding Module The decoder module reconstructs the obtained latent representation back into predictions of the original optical fiber motion, with the output being and nylon, designed to protect the optical fiber from the velocity information X (t) for each key point on the mechanical damage. This paper does not address the optical fiber at a specific time step. influence of the cladding on the refractive index of the fiber core but focuses mainly on the material character - 4.2 D ynamics Partial Differential Equations of Optical istics of the coating layer. Fiber Deformable Objects We used a UTM-5000 universal testing machine (accu- During the actual process of robotically pulling the opti- racy ±0.5%) to perform tensile tests on 20 samples of cal fiber, it becomes evident that the dynamic deforma - 0.6  m long fibers at a constant strain rate of 5  mm/min. tion of the fiber is highly complex. Predicting its shape Each optical fiber specimen is installed on the testing is crucial for successful operation, and the dynamic device with a pre-tension of 1 N to eliminate relaxation. behavior of the optical fiber can be described by stand - After 20 repeated tests, the elastic modulus was meas- ard dynamic partial differential equations. As a one- ured to be 2.5 GPa. The density was measured to be dimensional deformable object, the motion of the 1.2 g/cm using a density analyzer. optical fiber adheres to the continuum mechanic model. The material of the coating layer can be modeled as a In this context, the deformation of the fiber is smooth hyperelastic material, which exhibits highly nonlinear and predictable. characteristics in elastic deformation. For simplifica - In this paper, the mechanical model of the single- tion, this study adopts the Neo-Hookean model to simu- mode optical fiber (as shown in Figure  4) was simulated late the deformation of a 0.6  m long optical fiber with a by using the Abaqus software. The optical fiber is com - diameter of 1 mm. In the Abaqus software, the material posed of a core, a cladding layer and a coating layer, properties of the fiber are set as follows: elastic modu - and is a concentric cylinder, as shown in Figure  5. The lus E = 2.5 GPa , Poisson’s ratio ν = 0.38 , and density commonly used quartz optical fiber’s core and clad - ρ = 1.2 g/cm . The strain energy density function is ding are made of high-purity quartz glass with a small given by: amount of dopants. The dopants are used to ensure that the refractive index of the core is slightly higher � = C (I − 3) + (J − 1) , 10 1 (5) than that of the cladding. The coating layer is primarily composed of a specific ratio of acrylate, silicone resin, where, I = tr(C) is the first strain invariant (where C is the right Cauchy-Green tensor), and J = det(F ) is the determinant of the deformation gradient tensor F . The material parameters are determined by the elastic modu- lus E and Poisson’s ratio ν: E 6(1 − 2ν) C = , D = . 10 1 (6) 4(1 + ν) E In the simulation, the fiber is discretized into a struc - tured mesh of 200x20 (C3D8R elements). The bound - ary conditions are set as follows: all degrees of freedom are constrained on the left end, while a transverse con- Figure 4 Single-mode optical fiber diagram centrated force F = 1N is applied to the right end. The W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 7 of 12 the fiber, and A is the cross-sectional area. f denotes ext the external load, and N is the internal force vector. Among them, f can be represented by the speed ext R(t) at which the robot pulls the optical fiber at time t and the damping matrix D of the environmental model: f = DR(t). (8) ext And N includes the contributions of axial force and bending moment: N = EA ∈−EIκ , (9) where, E is the Young’s modulus, I is the moment of iner- ∂u(x,t) tia of the cross-section, the axial strain ∈= , and the ∂x ∂ u(x,t) curvature κ ≈ . ∂x Figure 6 Fiber optic displacement cloud map u Th s, when a force is applied at the end of the fiber, the dynamic partial differential equation governing the deformation of the fiber can be expressed as: 2 2 4 ∂ u(x, t) ∂ u(x, t) ∂ u(x, t) ρA = EA − EI + DR(t), 2 2 4 ∂t ∂x ∂x (10) ∂ u(x,t) where, EA represents the axial force of the fiber, ∂x consistent with the fundamental laws of elasticity, while ∂ u(x,t) −EI describes the fiber’s resistance to bending, ∂x effectively counteracting changes in curvature. 4.3 Loss Function for Physics‑Informed Fusion In the context of information-physics fusion, the con- struction of the loss function is a central concept of the PIGNN methodology. The loss function in PIGNN typi - Figure 7 Fiber stress cloud map cally consists of two components: data fitting loss and physical constraint loss, ensuring that the neural net- resulting displacement and stress contours are shown in work can learn from data while also adhering to physical Figures 6 and 7, respectively. laws. For the dynamic modeling of the deformable optical The displacement cloud image and stress cloud image fiber system, we can incorporate the partial differential obtained by simulation intuitively show the deformation equations governing the optical fiber into the physical distribution and stress concentration area of the fiber constraint component of the PIGNN, enabling shape under the action of load, which provides a basis for veri- prediction and learning of complex dynamic behaviors fying the accuracy of the hyperelastic material model. through the neural network. Based on these results, we combine simulation observa- Data fitting losses are used to ensure that PIGNN can tion with theoretical modeling to achieve quantitative fit real observational data. In this study, the data are analysis and prediction of the dynamic behavior of opti- derived from the optical fiber displacement measured by cal fibers. We further adopt the Euler-Bernoulli beam the sensor. Data fitting loss can be defined as: theory to establish the dynamic control equation of the optical fiber: 1 Loss = xˆ (t) − x (t) , (11) d i i i=1 ∂ u(x, t) ∂N ρA = + f . (7) ext ∂t ∂x where x ˆ (t) is the position of a key point on the fiber predicted by the model, x (t) is the actual observed data where, u(x, t) represents the displacement vector of the of the point, and N is the number of key points on the fiber at spatial coordinate x and time t; ρ is the density of fiber. We approach this loss by minimizing it to approxi - mate the experimental data. Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 8 of 12 In order to integrate the fiber deformation dynam - ˙ ˙ X(t) = f (X(t − m : t); R(t); θ). (16) PIGNN ics information into PIGNN, physical constraint loss is needed to ensure that the prediction results of graph neural network satisfy the partial differential equation 5 Experiments of fiber deformation dynamics. Using this equation as a 5.1 Experimental Setup constraint, the physical loss can be defined as: N 2 2 2 4 1 ∂ uˆ (x , t ) ∂ uˆ (x , t ) ∂ uˆ (x , t ) j j j j j j (12) Loss = ρA − EA − EI − DR(t) , 2 2 4 N  ∂t ∂x ∂x j=1 where u ˆ (x , t ) is the position vector of the optical fiber We conducted both simulation and real-world experi- j j predicted by the neural network, and N is the number of ments to validate the proposed method. Although we sampled points at different times and locations, through established a finite element model of the optical fiber, which the residuals of the system of partial differential we only used the derived partial differential equations equations are computed. as physical constraints for the PIGNN. Therefore, in the The total loss function is a weighted sum of the data bullet simulation environment based on Obi [26], we fitting loss and the physical constraint loss: modeled the optical fiber with its ends grasped by two robotic arm end-effectors. We tested a 3D task, where Loss = αLoss + βLoss . d p (13) the environmental dimension was 3 and the control input dimension n was 12. In the actual experiment, the opti- To dynamically balance the contributions of data cal fiber was placed on a table, with one end grasped by a fitting loss Loss and physical constraint loss Loss , d p UR5 robotic arm and the other end fixed. Consequently, weighting parameters α and β are defined as normal - the environmental dimension was 2, and the control ized ratios of loss gradient norms: input dimension n was 3. The shape of the optical fiber  � � � � ∇ Loss was represented by 13 key points, whose positions were  θ p � �  α = , � � obtained by measuring markers on the fiber using a cali - � � ∇ Loss + ∇ Loss +∈ θ d θ p (14) brated KinectV2 camera. In the experiments, both data  � � ∇ Loss θ d  � � β = , acquisition frequency and control frequency were set at � � �∇ Loss � + ∇ Loss +∈ θ θ p 10 Hz. The performance of the PIGNN model is evaluated where,  ∇ Loss and ∇ Loss represent the gradients θ d θ p based on the number of successful task completions, the of data loss and physical loss with respect to network average error in shape prediction, and the average time parameters θ respectively, and ϵ is a minimal value (set taken for predictions. The definition of the average error −8 to ϵ=10 ). When the magnitude of physical constraint in shape prediction is as follows: gradients significantly exceeds that of data fitting gradi - 13 13 ents, α is reduced to prevent physical constraints from 2 2 2 ˆ ˆ E = |d − xˆ | = (x − d ) + (x − d ) , dominating the optimization process excessively, and vice p i i i_x i_x i_y i_y i=1 i=1 versa. Through backpropagation of physical constraints, (17) we optimize the network model by simultaneously refin - where E represents the sum of squared errors for all ing data gradients and physical consistency via gradient key points, x ˆ denotes the predicted position of each key descent. During backpropagation, the update direction of point, and d indicates the target position of each key model parameters θ is determined by the weighted sum point. of composite gradients: The training data was collected in a simulated environ - θ ← θ − η(α∇ Loss + β∇ Loss ), θ θ p d (15) ment where the clampers randomly moved the ends of the optical fiber. First, we trained a graph neural network where η denotes the learning rate. The physical informa - to learn the initial deformation model. Then, we adjusted tion graph neural network f (·) leverages Tensor- PIGNN the computed preliminary deformation using the infor- Flow’s automatic differentiation to compute ∇ Loss . θ p mation from physical constraints. We initially tested the After updating θ through backpropagation, the model’s accuracy of both the data model and the physical mode- final corrected prediction is expressed as: ling on a specific optical fiber. Subsequently, we collected 10000 data points from the same optical fiber, with 70% W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 9 of 12 of the data designated as the training dataset, 15% as the Table 1 Performance of methods in optical fiber shape prediction task in simulation testing dataset, and the remaining 15% as the validation dataset. For testing and validation, we used the trained Methods Success rate Average task Average model to predict the shape of the optical fiber after 10 error (cm) task time (s) steps. GNN+Online Residual 76/100 0.553 10.324 5.2 Results Online learning 56/100 5.681 35.895 FKM+MPC 65/100 2.426 21.088 1) Simulation First, we tested the performance of Pure physical 96/100 0.239 248.451 the PIGNN model in the 3D shape prediction task Ours(w/o physical) 58/100 2.833 18.256 within the simulation, as shown in Figure 8. The con - Ours 90/100 0.251 7.562 trolled optical fiber was untrained, and we tested five cases with different target shapes. We also per - formed ablation studies, as detailed in Table  1. Our ber of successes and prediction accuracy, but it method significantly outperformed the compara - takes too long. The offline + online residual method tive methods in terms of success rates and average (GNN + Online Residual) performed relatively well task errors, even with less offline training data. The but achieved fewer successes than our approach. online learning method had the highest task time Our model without physical information (Ours(w/o since it required initialization of the Jacobian matrix physical)) achieved fewer successful trials and had by moving the ends of the deformable objects along a higher average task error. In contrast, the method each degree of freedom at each start. The combina - with physical constraints reached a 90% success rate tion of forward kinematics and model predictive con- and an average task error of 0.251 cm, second only to trol methods (FKM + MPC) exhibited higher aver- the pure physical model method. age task errors because it did not further update for 2) Real-world Experiments We evaluated these meth- the untrained deformable objects. The Pure physical ods in real 2D tasks. Using the same model as in the model method performs best in terms of the num- Figure 8: Simulation of optical fiber shape prediction, a, b, c, d, e represent the five target shape prediction tasks we have established Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 10 of 12 Figure 9 Shape prediction task testing 3, as indicated in Table  2, results show that the pure online learning method progresses slowly and une- venly, while the approach combining forward kin- ematics and model predictive control (FKM+MPC) is fast but less accurate. The offline + online residual method (GNN + Online Residual) achieves similar success rates and average task times as our model. The average task error of the pure physical model method is the lowest, but the time consumption is also the longest. After removing the physical infor- mation (Ours(w/o physical)), our model still per- Figure 10 Case 1 (0.6 m fiber optic) formed poorly in the actual experiments, while our model with physical constraints was superior to the offline + online residual method in the average task error. The actual experimental results prove that our method has the highest number of successes among other methods except for the pure physical model and achieves the lowest average task error. The physi - cal constraints of optical fibers make the prediction faster and more accurate. Although the average task error of the pure physical model is the lowest and has high accuracy, its prediction speed is the slowest and does not meet the requirements of real-time control of optical fibers. Figure 11 Case 4 (1 m fiber optic) The main advantages of our method are reflected in several aspects: First, PIGNN effectively captures the complex nonlinear characteristics of the optical fiber simulation, we did not collect real-world data for during the winding process by incorporating physical training. Considering that one end of the optical fiber laws as constraints into the model, thereby enhancing is fixed during the winding process, we employed a the model’s robustness and reliability. Second, PIGNN single arm to perform operations and conducted five utilizes graph neural networks to extract global dynamic different feasible winding shape tests on two types of information, performing better than traditional data- optical fibers: one with a length of 0.6 m and a diam - driven methods under conditions of limited data. The eter of 1 mm, and another with a length of 1 m and a experimental results show that it is second only to the diameter of 1  mm, as shown in Figure  9. Figures  10 Pure physical model method in terms of task success rate and 11 visualize the prediction processes for both and average task error. Moreover, PIGNN introduces cases. Due to the control input dimension being only W ang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 11 of 12 Table 2 Performance in real-world 2D shape prediction tasks Methods case1(0.6 m fiber optic) case4 (1 m fiber optic) Success rate Average task Average task Success rate Average task Average error (cm) time (s) error (cm) task time (s) GNN+Online Residual 9/10 1.211 10.114 9/10 1.782 14.324 Online learning 8/10 7.442 22.895 7/10 6.311 25.432 FKM+MPC 8/10 2.997 12.047 8/10 3.626 16.758 Pure physical 10/10 0.985 384.532 10/10 1.246 667.289 Ours(w/o physical) 6/10 2.760 15.776 6/10 3.818 16.442 Ours 9/10 1.188 10.885 10/10 1.551 13.572 physical constraints as the loss function. In tasks involv- optical fibers and their surroundings, aiming to pro - ing complex deformation of optical fibers (such as case4), vide smarter solutions for optical fiber manipulation in the predicted deformation error is small, and it has the practical industrial scenarios. fastest prediction speed among all methods. In summary, PIGNN’s comprehensive performance in multiple tasks 1) Multi-step Operations (Grasp and Release Required) is superior to other methods, which proves its effective - Real-world optical fiber manipulation tasks may ness and superiority as an optical fiber shape prediction require multi-step shaping, necessitating changes model, and provides an efficient and accurate solution for in the grasping points during operation. Identifying intelligent disk fiber tasks. optimal grasping points for the optical fiber presents a challenging research problem. We plan to utilize Transporter Networks [27] to learn grasping points 6 Conclusions from images and integrate them with our manipula- In this paper, an innovative method based on physical tion approach. information graph neural network (PIGNN) is proposed 2) Optical Fiber Winding Environments Optical fiber to predict the shape of deformable linear objects, such as manipulation tasks predominantly involve winding optical fibers. Compared with the traditional data-driven processes and must consider complex environments. method, this method combines the powerful expression Model-based approaches are rarely proposed in the ability of graph neural network with the constraint of literature. Mitrano  et al. [28] introduced a method physical law, and provides a new perspective and solu- to train classifiers for determining effective positions tion for the deformation problem of deformable linear for offline learning models and employed learning objects. The graph neural network (GNN) is introduced strategies to recover robot functionality when the to learn the initial deformation model of the fiber. During model was unreliable. However, this method does the training process, the physical prior knowledge of the not adequately address scenarios where the optical dynamic deformation of the fiber is incorporated into the fiber needs to interact with the environment. We will loss function as a constraint and fed back to the network pursue this as a direction for future research. model, thus ensuring that the shape of the fiber is gradu - ally close to the real target shape. Acknowledgements The experimental results show that under the condi - Not applicable. tion of data scarcity, our method performs well in both prediction accuracy and stability, especially in the task Author contributions MW and JW determined the overall framework and central idea of the article; of complex deformation such as bending and twisting of MW was responsible for all experiments and analysis, and wrote and checked optical fiber. In comparison to conventional methods, the the manuscript; JZ and JW assisted with the manuscript; XL and GL were PIGNN model shows marked improvements in execution responsible for a part of experiments. All authors read and approved the final manuscript. time and prediction error, with strong performance in real-time predictions suggesting promising applications. Funding Future research will further extend the application of Supported by the Fundamental Research Funds for the Central Universities (Grant Nos. 2232024Y-01, LZB2023001), DHU Distinguished Young Professor this method in more complex three-dimensional envi- Program, National Natural Science Foundation of China (Grant No. 52275478), ronments and explore its adaptability and robustness AI-Enhanced Research Program of Shanghai Municipal Education Commission in dealing with the interactions between deformable (Grant No. SMEC-AI-DHUY-05) Wang et al. Chinese Journal of Mechanical Engineering (2025) 38:161 Page 12 of 12 Availability of data and materials [18] J Zhu, D Navarro-Alarcon, R Passama, et al. Vision-based manipulation of Not applicable. deformable and rigid objects using subspace projections of 2D contours. Robotics and Autonomous Systems, 2021, 142: 103798. [19] R Y Ming, H Z zhong, X Li. Shape control of deformable linear objects Declarations with offline and online learning of local linear deformation models. IEEE International Conference on Robotics and Automation (ICRA), Philadelphia, Competing Interests PA, USA, May 23-27, 2022: 1337-1343. The authors declare no competing financial interests. [20] Y Z Li, J J Wu, R Tedrake, et al. Learning particle dynamics for manipulating th rigid bodies, deformable objects, and fluids. 7 International Conference on Learning Representations (ICLR), New Orleans, LA, USA, May 06-09, 2019. Received: 24 December 2024 Revised: 5 June 2025 Accepted: 11 June [21] H Wakamatsu, K Takahashi, S Hirai. Dynamic modeling of linear object deformation based on differential geometry coordinates. IEEE Interna- tional Conference on Robotics and Automation, Barcelona, Spain, April 18-22, 2005: 1028-1033. [22] F F Khalil, P Payeur. Dexterous robotic manipulation of deformable objects with multi-sensory feedback - a review. Robot Manipulators Trends References and Development. United States: INTECH Open Access Publisher, 2010: [1] X Li, X Su, Y Gao, et al. Vision-based robotic grasping and manipulation 587-621. of USB wires. IEEE International Conference on Robotics and Automation [23] G F Pang, L Lu, G E Karniadakis, et al. fPINNs: Fractional physics-informed (ICRA). Brisbane, QLD, Australia, May 21-25, 2018: 3482-3487. neural networks. SIAM Journal on Scientific Computing, 2019, 41(4): [2] L Cao, X Li, P T Phan, et al. Sewing up the wounds: A robotic suturing A2603-A2626. system for flexible endoscopy. IEEE Robotics & Automation Magazine, 2020, [24] S Z Cai, Z C Wang, S F Wang, et al. Physics-informed neural networks 27(3): 45-54. for heat transfer problems. Journal of Heat Transfer, 2021, 143(6): [3] R Lagneau, A Krupa, M Marchal. Automatic shape control of deformable 060801-060815. wires based on model-free visual servoing. IEEE Robotics and Automation [25] Z P Mao, A D Jagtap, G E Karniadakis. Physics-informed neural networks Letters, 2020, 5(4): 5252 - 5259. for high-speed flows. Computer Methods in Applied Mechanics and Engi- [4] Z Hu, T Han, P G Sun, et al. 3-D deformable object manipulation using deep neering, 2020, 360: 112789. neural networks. IEEE Robotics and Automation Letters, 2019, 4(4): 4255-4261. [26] V M Studio. Obi: Unified particle physics for Unity 3D, 2019. http:// obi. [5] C Morley, R V Patal. Steering of flexible needles using an LSTM encoder with virtu almet hodst udio. com. model predictive control. 2022 2nd International Conference on Robotics, [27] D Seita, P Florence, J Tompson, et al. Learning to rearrange deformable Automation and Artificial Intelligence (RAAI), Singapore, December 09-11, cables, fabrics, and bags with goal-conditioned transporter networks. 2022, 99-104. IEEE International Conference on Robotics and Automation (ICRA), Xi’an, [6] A Nealen, M Müller, R Keiser, et al. Physically based deformable models in China, May 30-June 05, 2021: 4568-4575. computer graphics. Computer Graphics Forum, 2006, 25(4): 809-836. [28] P Mitrano, D McConachie, D Berenson. Learning where to trust unreliable [7] F S Sin, D Schroeder, J Barbic. Vega: Non-linear FEM deformable object models in an unstructured world for deformable object manipulation. simulator. Computer Graphics Forum, 2013, 32(1): 36-48. Science Robotics, 2021, 6(54): eabd8170. [8] E Yoshida, K Ayusawa, I G Ramirez-Alpizar, et al. Simulation-based optimal motion planning for deformable object. IEEE International Workshop on Advanced Robotics and its Social Impacts (ARSO). Lyon, France, June 30-July Meixuan Wang born in 1999, is currently a master’s candidate at 02, 2015: 1-6. the College of Mechanical Engineering, Donghua University, China. Her [9] G Frugoli, A Galimberti, C Rizzi, et al. Discrete element approach for research focuses on machine vision in intelligent manufacturing non-rigid material modeling. Robot Manipulation of Deformable Objects. systems. Advanced Manufacturing, London: Springer, 2000: 29-41. [10] M Raissi, P Perdikaris, G E Karniadakis. Physics-informed neural networks: A deep learning framework for solving forward and inverse problems Junliang Wang born in 1991, is currently an associate researcher involving nonlinear partial differential equations. Journal of Computa- at Institute of Artificial Intelligence, Donghua University, China. He tional Physics, 2019, 378: 686-707. received his PhD from Shanghai Jiao Tong University, China. His [11] W Yan, A Vangipuram, P Abbeel, et al. Learning predictive representations research focuses on smart manufacturing and robotics, computer for deformable objects using contrastive estimation. Proceedings of the vision, and big data intelligence. Conference on Robot Learning. 2020, 155: 564-574. [12] W B Zhang, K Schmeckpeper, P Chaudhari, et al. Deformable linear object Jie Zhang born in 1963, is currently a professor and the Executive prediction using locally linear latent dynamics. 2021 IEEE International Dean at Institute of Artificial Intelligence, Donghua University, China. She Conference on Robotics and Automation (ICRA), Xi’an, China, May 30-June received her PhD from Nanjing University of Aeronautics and Astronau- 05, 2021: 13503-13509. [13] Y Yang, J A Stork, T Stoyanov. Learning to propagate interaction effects for tics, China. Her research interests include smart manufacturing and Modeling Deformable Linear Objects Dynamics. 2021 IEEE International robotics, big data intelligence, machine cognitive learning, and com- Conference on Robotics and Automation (ICRA), Xi’an, China, May 30-June plex system modeling and control. 05, 2021: 1950-1957. [14] A Ajay, J Wu, N Fazeli, et al. Augmenting physical simulators with stochas- Xinting Liao born in 1997, is currently a PhD candidate at College tic neural networks: Case study of planar pushing and bouncing. IEEE/RSJ of Mechanical Engineering, Donghua University, China. She received International Conference on Intelligent Robots and Systems (IROS), Madrid, her Master’s degree from South China Agricultural University, China. Spain, October 01-05, 2018: 3066-3073. Her research focuses on pattern recognition and quality control in [15] A Kloss, S Schaal, J Bohg. Combining learned and analytical models for intelligent manufacturing systems. predicting action effects from sensory data. The International Journal of Robotics Research, 2022, 41 (8): 778-797. [16] D Navarro-Alarcon, Y H Liu, J G Romero, et al. Model-free visually servoed Guojin Li born in 1984, is currently an engineer at the Central deformation control of elastic objects by robot manipulators. IEEE Trans. South Architectural Design Institute Co., Ltd., China. His research inter- Robotics, 2013, 29 (6): 1457-1468. ests include deep learning, computer vision, and natural language [17] J Zhu, B Navarro, P Fraisse, et al. Dual-arm robotic manipulation of flexible processing. cables. IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Madrid, Spain, October 01-05, 2018, 479-484.

Journal

Chinese Journal of Mechanical Engineering – Springer Journals

Published: Aug 20, 2025

Keywords: Deformable linear objects; Fiber; Physics-informed graph neural network (PIGNN); Shape prediction

There are no references for this article.