Can I use thus card {Unsubstantiate} to return a spell I just cast?
Returning the spell to your hand will remove the spell itself from the stack and then back to your hand. This can however be used to abuse any on cast effects as those happen when you cast the spell and it resolves. For example if you {Unsubstantiate} an {Ulamog, the Ceaseless Hunger} from the stack, the owner of the Ulamog still gets the exile two permanents trigger.
Wouldn't work with the gearhulks they specify when they enter the battlefield. Ulamog is when you cast him.
You can also use it on a spell of yours that would be countered. Or stuff with Cascade, specially if you are playing Yidris EDH for 2 triggers, plus another later. If you combine it with stuff like {Oblivion Sower} or Ulamog as mentioned, you get crazy value!