Adding Batch package

[tmatsuo]

Adding the Batch package for asynchronously execute some jobs in batch.

The basic design is described at: https://wp.gaeflex.ninja/2017/03/02/asyncbatchrunner-design-proposal/

[jdpedrie]

I wonder if the BatchDaemon could be run as part of the composer google-cloud CLI command we already have? It's currently used just for development stuff, but there's not really a reason we couldn't expand that.

[tmatsuo]

@jdpedrie That's a good idea! Can we install vendor/bin/google-cloud when installed by composer?

[tmatsuo]

@jdpedrie On the second thought, I think it's too much to include symfony/console as production dependency, just for the daemon. I would create an entry point script for the daemon and install it in vendor/bin.

[jdpedrie]

Good point!

[tmatsuo]

@jdpedrie @dwsupplee I think it's ready for a real review. PTAL

Some examples for using this library.

BatchLogger (need to put it somewhere autoloadable)

<?php

namespace Google\Cloud\Core\Logger;

use Google\Cloud\Logging\LoggingClient;

/**                                                                                                                                                                
 * A logger that batches logs.                                                                                                                                     
 */
class BatchLogger
{
    /**                                                                                                                                                            
     * Send the given log entries.                                                                                                                                 
     *                                                                                                                                                             
     * @param array $items                                                                                                                                         
     */
    public function sendEntries($items)
    {                                                                                                                                           
        $c = new LoggingClient();
        $logger = $c->logger('my-log', [
            'resource' => [
                'type' => 'gcs_bucket',
                'labels' => [
                    'bucket_name' => 'my_bucket'
                ]
            ]
        ]);
        try {
            $logger->writeBatch($items);
        } catch (\Exception $e) {
            fwrite(STDERR, $e->getMessage() . PHP_EOL);
            return false;
        }
        return true;
    }
}

The code for registering the job:

use Google\Cloud\Core\Batch\BatchRunner;
use Google\Cloud\Core\Logger\BatchLogger;

if (count($argv) === 2) {
    $num = intval($argv[1]);
} else {
    $num = 1;
}

$batchLogger = new BatchLogger();

$r = new BatchRunner();
$r->registerjob(
    'logging',
    array($batchLogger, 'sendEntries'),
    array(
        'workerNum' => $num,
        'batchSize' => 1000
    )
);

The code for submitting the item:

use Google\Cloud\Core\Batch\BatchRunner;
use Google\Cloud\Logging\LoggingClient;

$r = new BatchRunner();
$c = new LoggingClient();
$logger = $c->logger('my-log', [
    'resource' => [
        'type' => 'gcs_bucket',
        'labels' => [
            'bucket_name' => 'my_bucket'
        ]
    ]
]);

if (count($argv) == 1) {
    $num = 10000;
} else {
    $num = intval($argv[1]);
}

//$message = str_repeat('x', 8192);
$message = 'log message';

$start = microtime(true);
for ($i = 0; $i < $num; $i++) {
    $r->submitItem('logging', $logger->entry($message));
}
$end = microtime(true);
printf('%d logs sent in %f seconds, at %f LPS' . PHP_EOL, $num, $end - $start, $num / ($end - $start));

For the logging client to work, you may need to set an envvar for auth:

export GOOGLE_APPLICATION_CREDENTIALS=/some/where/secret.json

In the first terminal, you can run the daemon:

bin/google-cloud-batch-daemon

Then export an envvar:

export IS_BATCH_DAEMON_RUNNING=true

and register the job and run the submit code. It can send ~20k logs/seconds with 1000 batch and 10 child processes :)

[tmatsuo]

Thanks @jdpedrie for the review! Is there anything left?

[dwsupplee]

I'd like to take a look as well, I have some time slated for tomorrow. Sorry for the hold up we've been a bit busy :).

[tmatsuo]

FYI, the BatchLogger class (the client of this Batch package) is available at:
https://github.com/tmatsuo/google-cloud-php/tree/batch-logger

These libraries perform really really well and I'm so excited :)

@dwsupplee
I'd appreciate it if we could proceed on this PR.

[dwsupplee]

This is a fantastic addition, thank you @tmatsuo. 👍

[tmatsuo]

@dwsupplee Thanks for your review! PTAL

[tmatsuo]

@dwsupplee

I needed to revert the change in BatchDaemon for performance reasons. Please see the commit comment for more detail.

[dwsupplee]

LGTM :)