Yes, most likely it drifts, or you fall asleep for an instant and it can move, and then the thing “appears”.
Happens all the time, even to non-sorcerers!
But they shake their head, rub their eyes, and make it go away.
That’s one thing you stop doing when you get further along.
“Making stuff go away” by doing a double take if something is odd, and turning quickly to check it out and prove it wasn’t anything you don’t understand.
Later, you just keep walking and don’t turn to look at something impossible, standing in your path.
As to how it gets the initial movement, Tensegrity is designed to do that, if you’re silent.
But “the right way of walking” can also cause it to move.
And if you force yourself silent for a long time, you’ll doze off for sure, even if you don’t actually fall over and go to sleep.
You’ll send up “sleep walking”, and suddenly you’ll be looking at stuff through the eyes of your dreamer, despite being awake.