Low-priority task reliability #314

Activity

1. brnleehng commented on Oct 1, 2018

   

   Brian Hoang (brnleehng)

   on Oct 1, 2018

   Collaborator

   More actions

   Hi Simon,

   1. Yes. When a task gets interrupted by a preempted VM, Batch will detect this and requeue the task to run again. The task will then be scheduled whenever a VM is available. Then the task will be re run again. This means the task has to be idempotent.

   • What do you mean by restart the job and where it previously crashed? Do you mean you start the foreach loop at an certain iteration point and it'll work?
   • Is the iteration of the foreach loop failing because of the preempted VM? Are there logs for the task failure that I can investigate?

   2. Are you running your clusters at a fix amount of VMs? Perhaps you can run it with autoscale with VMs scaling down to 1 for the VM.

   How much data are the task results? If the disks are full, the VM runs into issues of changing its state.

   I would recommend disabling the merge task and run a separate job to merge the results. Are you running with chunksize greater than 1? This will get you partial merge results. If the results are quite large or too many, the R session in the merge task can hit a memory issue since it has to hold all of the results in memory for merging anyways.

   3. We can add more feedback on the merge task. Possibly write more logs to the stdout so users can see what the state of the merge task. However, this will slow down the merge task even more.. One benefit of using long running jobs instead of an active R session is you can submit all your jobs at once instead of waiting for each job to be done so then you don't need to wait for the 'hung' merge task.

   Is the hung merge task getting preempted then it gets rescheduled and then becomes hung again?
   Is each job dependent of each other?

   Unfortunately, I think this is the nature of low priority VMs. The maximum lifetime of the low priority VMs are generally 24 hours.

   You can reach out to razurebatch@microsoft.com for more detailed situations. I would be interested in learning more about your workflow scenario.

   Thanks for your feedback,
   Brian
2. simon-tarr commented on Oct 1, 2018

   

   Simon (simon-tarr)

   on Oct 1, 2018

   Author

   More actions

   Hi Brian, thanks very much for the detailed response - it's very much appreciated.

   I can't answer many of your questions immediately but you've given me plenty to think about. I'll run some tests and calculate how much memory etc the jobs are taking up. They're reasonably large but probably not so large that the cluster sizes I'm dealing with can't handle it. I am using a chunksize >1 (chunksize is always 130 in my current use case).

   With regards to your first bullet point; that is what I meant, yes. Essentially, the crashing isn't anything to do with any particular model. They will all run eventually if I restart the cluster and try again. So it's not like I'm working with a faulty model that can't run given the input parameters (e.g. a model that can't converge for whatever reason). I believe the crashing is an Azure problem - whether this is memory, pre-empted nodes etc etc.

   If you could let me know the exact logs that you'll need then I can take some time to grab these the next time it crashes.

   If it's not too much trouble, could you please explain how I can disabling the merge task and run a separate job to merge the results? Is this documented somewhere? I'd be happy to give this a go.

   Finally, the reliability has been pretty decent until a few weeks back which makes me think that something has changed on the backend...I rarely encountered this issues a month ago. Now every other job seems to crash for one reason or another. I know that's a bit of a hand-wavey statement in lieu of error logs but, subjectively, something has defintely changed.
3. brnleehng commented on Oct 3, 2018

   

   Brian Hoang (brnleehng)

   on Oct 3, 2018

   Last edited by brnleehng

   Collaborator

   More actions

   The merge task processes each file one by one, adding it to a list. If you are submitting jobs one by one, then the merge task will block the other job submissions, wasting core hours.

   If the merge task appears to be hung, can you give me the job id so I can check our event logs? I'll see if the service made any changes..
   Aren't these tasks idempotent? So you should let the tasks get re queued and be assign to another VM
   What's the average duration of these tasks?

   Here's a sample I wrote of disabling the merge task and combining the results locally.

   setAutoDeleteJob(value = FALSE)
   setCredentials("credentials.json")
   
   cluster <- makeCluster("cluster.json")
   registerDoAzureParallel(cluster)
   results <- vector("list", 3)
   for(i in 1:length(results)){
     start = results * i
     end = start + 3
     
     results[[i]] <- foreach(i = start:end, .options.azure = list(enableCloudCombine = FALSE,
                                                            wait = FALSE,
                                                            job = paste0("job"-i))) %dopar% {
       
     }
   }
   
   # Wait for jobs to complete
   
   for(i in 1:length(results)){
     results <- getJobResult(results[[i]])
   }

   Thanks,
   Brian
4. Simon (simon-tarr)

   closed this as completedon Oct 17, 2018