No, without a court order dictating when each parent has custody of the kids, there is nothing that would require you to give the children back to your spouse whenever they demand that you do so. However, during custody disputes, the courts want the parents to set aside their differences and work together for the betterment of the children. Therefore, it would be a good idea to try and come to some sort of arrangement with your ex about when to drop off the kids that is agreeable to both of you.