Oza

First Principles in Motion

I built a working digital twin in a couple of days: a Beckhoff controller running a pair of ABB delta robots that pick food items off a moving belt and drop them into moving boxes. Yes, this is the famous conveyor tracking problem well studied and provided by many well-known manufacturers in the realm of industrial robotics (see here and there). It finds applications in food & beverages factories and packaging of fast moving consumer goods.

Salient feature of this blog is (soft) real-time communication between the PLC and digital plant. One can say this is a Hardware-in-the-loop (HIL) involving PLC and robot for the classic conveyor tracking problem. For the academic interest of doing it all as independently as possible, I used Isaacsim library by NVIDIA. The choice of PLC was motivated by suitability of Beckhoff for Windows platform. As for NVIDIA, it was not just the physics engine but also the communication interfaces between a Beckhoff runtime and Nvidia runtime that drove the architectural choice. The STEP files for the ABB robots are on their library page.

On why do such an activity, a nice explanation can be viewed here and there.

Here is how mine renders.

Is this really new?

Short answer is a 'no'. Every ingredient exists in public domain on its own. A simulator ships a conveyor tool, but it picks from fixed poses, not by matching a moving product's speed [1]. Vendors have trained grippers for items on fast belts, but that is teaching a camera to see, not tracking control [2]. A real Beckhoff controller can already talk to this simulator through an open bridge [3]. Conveyor tracking is decades old on real robots, just not inside this simulator [4]. And running a closed-loop delta robot here is awkward, with no public example to take inspiration from [5].

Rigorous academic review of literature is not what I mainly write blogs for :) but to be honest the claim is narrow: closing a real Beckhoff controller around a high fidelity simulator to have a delta robot doing speed-matched conveyor-tracking pick-and-place seems to have no public example before it, even though every piece exists on its own. The new part is the combination.

Besides the present blog still falls short on the offline program front. It does not match the robot behaviour via exact robot programming artifacts. I use plant integration techniques native to Isaacsim via the first principles of physics and not MoveL commands of ABB robots, for example.

I built it by directing an AI coding agent (Claude Code). I set the rules and made the calls. It wrote the code and I ran it on the rig (another workstation with NVIDIA GPU to handle the heavy physics of fast Delta robots). What follows is less about the robot and more about how I worked, to produce a high fidelity digital twin. It successfully answered this question that I kept posing to myself.

How do you get a real factory controller to believe it is running a machine that does not exist, without quietly fooling yourself?

The code lives here.

Decide the rules before you write code

This is something the state of the art of AI is not yet able to master and is central to the current debates on what the AI will be able to do in future or what it should be able to do today. A human is needed to make sense of and build plausible physical artifacts, even if it is in a digital realm.

I did not start with code. I started by listing what should never be violated. Not tips or habits, just the few truths that, if you break one, the result is wrong no matter how good it looks. Here they were simple: A set of first principle based rules of motion. For example, a grab is real only if the tool and the part share the same place, non-zero contact force and speed. Every part is counted, and none disappears. Not surprisingly, these primitives are needed to get the AI to generate plausible physics related code.

Only from those rules did I write the tests, the kind you can pass or fail and check yourself. For example, a grab counts only when position and speed match. Parts in equals parts picked plus parts that got by. A control loop under two milliseconds. Every test pointed back to a rule.

The tricky part of working with an AI is defining what "correct" means, exactly enough that neither you nor AI can drift. Draw the ruler before you start, and later, when you widen a window or bump the belt speed, you run the same tests every time. The ruler never moves. This method leads to a solid contract between the human user and AI.

Identify non negotiable Principles to be followed

Identify evals and tests tied to each principle

Let AI and human actions be cross-checked via evals

For our current exercise, it gave rise to this constitution which is enforced by these tests.

Mock it first, then fill it in

First, I built a mock replica, then swapped it for the real one, one layer at a time. This applies to spatial and time behavior for the robots. For example, before a line touched the real PLC controller, a mock controller in Python behaved exactly like the real one would and drove a plain-Python plant with no graphics. That let us settle the behavior on a laptop in milliseconds. Only once that mock was our gold standard did I translate it, line for line, into the real controller.

A mock is actually the main promise. When the real controller finally drove the cell and looked identical to the mock one, that was not luck. The mock one had already said what "correct" looked like. The one time they differed, finding the reason took minutes instead of hours. This is the crux of the 'Principle to Evals' contract.

An example of a mock PLC within Isaacsim is here, whereas a mock plant for the PLC to talk to is there. The real PLC code lives here and it does need some knowledge on Beckhoff's ways of programming.

The parts, and the clock

The controller main program is small on purpose. Each robot is one small state machine: claim a part, follow it at belt speed, grip after a short lock, carry, commit to a box, come down, release, go home. The upstream robot takes its share; the downstream one is greedy and grabs whatever slips past. One enable switch freezes everything, just like a user of a machine would like to do.

The interesting part is time. A controller runs on a hard, steady beat (the famous ADS protocol by Beckhoff). A simulation in Isaacsim, on the other hand, only moves forward when you hand it a step of time, a dt, the infinitesimal step for the integral calculus of Isaacsim. I let the simulation read the controller's clock (via pyads library) instead of guessing or letting it run on simulator's system clock:

dt = (plc_time_ns - prev_plc_time_ns) / 1e9
if dt > 0:
    plant.step(dt)      # advance ONLY when the PLC clock actually ticked

In real scenario, the plant integrates itself and 'plant.step(dt)' is not needed in real world! In the absence of a real plat, I can get as close to it as possible using standard numerical integration tricks (which is a little out of scope here).

One clock, that of the controller's, drives both sides and corrects for the small random delays in when items land. The round trip (or the loop sampling time) is 2 milliseconds, with a quarter of a millisecond of jitter in PLC runtime. Jitter comes in because Isaacsim is not running on an RTOS. That is not for show. It is important to understand why these numbers matter. The delay in the loop is your aiming error budget: a part moving at a fifth of a meter per second, read two milliseconds late, is off by less than half a millimeter, inside grip tolerance with room to spare. A real cell runs the same controller over a hard fieldbus that is tighter and steadier still, so the timing carries over. Two milliseconds is a green light. Ten seconds sampling time with five seconds of jitter would kill the project.

The clock is the contract. Positions and speeds are easy. The signal that has to be right, is time.

A plant that flows and a mind that ticks

The most important aspect was whether I was modeling the right thing. The obvious answer to keeping two things in sync is to use a descrete time version of the plant and in turn use one shared clock to step the dynamics. But here is the thing: a real plant flows. Most physical plants (like robots) are analogue - not digital. They move in continuous time, with no clock and no ticks. Wrapping it in a master clock forces digital rigidity onto something that does not have digital nature. This is the famous digital divide studied in control systems.

Seen that way, the 'PLC ticking Isaacsim' is the honest shape of sampled control. In ultimate reality a digital controller will drive an analogue plant.

And it is not perfect. Robot B still lets a food item or two get by while holding one and waiting for a box. I would rather have that honest limit than a demo that hides it. It ran. Both robots picked. The controller was in charge. Above all, conveyor tracking worked.

Taking it to the floor

A twin only earns its keep as the on-ramp to real hardware. What stays, what goes, what shows up?

What stays. The whole controller program, unchanged. That is the point of hardware-in-the-loop: the controller that ran the mock cell runs the real one. The tag map and the sampled-control discipline stay. The simulator stays too, but its job flips, from being the plant to running alongside the real line as a shadow. Of course, the control logic can be made superior, e.g., bringing in feed forward velocity mode and its fusion with position control mode and similar. One can also bring ready-made robotics libraries by Beckhoff or other PLC vendors to feed singals to the Digital Twin. It would be a nice next exercise to do that.

What goes. In the twin the plant decides whether a grab lined up; on the bench, real physics, a vacuum gripper, and cameras decide. The soft time base goes, replaced by a hard fieldbus. The Python stand-in bridge drops out.

What shows up. The expensive half. Two real delta robots and drives, a hard fieldbus with a shared clock, a vision system, belt encoders, safety interlocks, and calibration to make the robot's frame and the camera's frame agree to a fraction of a millimeter.


There is a bit of an irony hiding behind any digital twin. The creator of the twin will benefit from having some prior experience of building the real world version. The real value belongs to the extension of one's own knowledge and not in verification of what is already known.



Next version will target issuing joint commands twinning with real motors of a robot.

References

  1. NVIDIA Isaac Sim, Conveyor Belt Utility and pick-and-place tutorials. docs.isaacsim.omniverse.nvidia.com (conveyor utility; pick from static poses).
  2. Soft Robotics / Oxipital, mGripAI trained in NVIDIA Isaac Sim, synthetic-data gripper training for conveyor items. blogs.nvidia.com / therobotreport.com.
  3. Loupe, Omniverse Beckhoff Bridge Extension, open-source TwinCAT to Isaac Sim over ADS. github.com/loupeteam/Omniverse_Beckhoff_Bridge_Extension.
  4. Visual conveyor tracking / pick-on-the-fly. RoboDK conveyor tracking; FANUC line tracking; IEEE, Visual conveyor tracking for pick-on-the-fly robot motion control.
  5. Isaac Sim / PhysX closed-loop articulations. Closed kinematic loops need guide-joint or exclude-from-articulation workarounds; no public IRB 360 / FlexPicker pick-and-place example found.