Introduce batch publisher by dwsupplee · Pull Request #534 · googleapis/google-cloud-php

dwsupplee · 2017-06-13T00:38:36Z

Closes: #458

This PR introduces a batch publisher, building on top of @tmatsuo's very powerful batch work.

At this stage, it would be awesome to get some general feedback as to the approach taken (and thoughts on the caveats below).

/cc @bshaffer @chingor13 @jdpedrie @mdietz94 @michaelbausor @tmatsuo

Example Usage

Starting up the daemon:

Please note, this is optional - if you choose to not use the daemon the batched messages will be published at the end of the request lifecycle via register_shutdown_function():

export IS_BATCH_DAEMON_RUNNING=true
./src/Core/bin/google-cloud-batch daemon

Sample script:

use Google\Cloud\PubSub\PubSubClient;

$pubsub = new PubSubClient();
$pubsub->topic('the_topic_of_your_dreams')
    ->batchPublisher()
    ->publish([
        'data' => 'What a great message!'
    ]);

Notes on the implementation

A suggested approach was add a publishAsync method to Topic. I ran into a few issues with this:
- Passing in options related to the batch job on each call does not seem ideal. Encapsulating the options inside of a class helps to make them more reusable. It would be possible to pass the batch options along through the Topic constructor, but there are a number of ways users can get a Topic (PubSubClient::createTopic(), PubSubClient::topic(), PubSubClient::topics(), Snapshot::topic()) and it seemed cleaner to me to have one place where you supply the options.
- Down the line, we plan to add Async variations of methods that trigger network requests across the veneer libraries (Asynchronous request support #31). If possible, we would like to reserve the name publishAsync for the use case where a user does not require batching but would like the benefit of concurrent network requests.
The use of the BatchPublishContainer could be mimicked in Google\Cloud\Logging\PsrLogger, allowing us to not worry about needing to serialize the entire PsrLogger for the daemon to process incoming data. This could open us up to having a shouldBatch (or similar) option on the PsrLogger, so we only have one class users need to worry about interfacing with. This approach also has the benefit of reducing the public API surface we need to maintain in the PsrBatchLogger (as we could remove the sendEntries public method). Please let me know what you think @tmatsuo.
Attempting to process the jobs in a cron job wasn't efficient as the minimum timeframe a cron job can be scheduled for is 1 minute. If anyone has any further experience with this though, I'm all ears!

Caveats

The daemon relies on shared memory and PCNTL, extensions which can't always be guaranteed to be available. The latter has no windows support. Additionally, shared memory won't scale well across multiple nodes.

With that said, I do feel like this will work great for the good share of our use cases.

@mdietz94 Do you think these restrictions are acceptable for us moving forward with beta? I think it would be worth investigating a more robust message queue to handle batch processing - but that will take us more time. We could provide methods to interface with redis, mysql, etc., and look at including a daemon that is built in something a bit more portable than we have currently.

Thanks for any thoughts!

Todos

- Tests
- Ability to flush queued messages on demand

tmatsuo · 2017-06-14T01:23:15Z

The use of the BatchPublishContainer could be mimicked in Google\Cloud\Logging\PsrLogger, allowing us to not worry about needing to serialize the entire PsrLogger for the daemon to process incoming data. This could open us up to having a shouldBatch (or similar) option on the PsrLogger, so we only have one class users need to worry about interfacing with. This approach also has the benefit of reducing the public API surface we need to maintain in the PsrBatchLogger (as we could remove the sendEntries public method). Please let me know what you think @tmatsuo.

I think that means introducing something like BatchLoggerContainer, and think it as though it's a private class, and the @internal annotation makes it so. Sounds reasonable to me.

Sign in to view

+        }
+        $end = microtime(true);
+        if ($this->debugOutput) {
+            printf(


dwsupplee · 2017-06-19T13:26:18Z

If there is anything regarding this approach we would like to discuss further please let me know, otherwise I'm going to finish up the rest of the implementation/tests now.

mdietz94 · 2017-06-19T16:13:20Z

SGTM

…

On Mon, Jun 19, 2017 at 9:26 AM, David Supplee ***@***.***> wrote: If there is anything regarding this approach we would like to discuss further please let me know, otherwise I'm going to finish up the rest of the implementation/tests now. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#534 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABvXaltvdAY1I1ET0VERUHtMXQd1pqVxks5sFnb8gaJpZM4N32c-> .

dwsupplee · 2017-06-27T17:23:25Z

All set and ready for review! If this looks good to everyone, this will allow us to move forward with getting the batch branch in master and published in our next release 😄.

A few changes worth noting:

@tmatsuo, I removed the PsrBatchLogger in favor of a single PsrLogger class which will support both batch and sync calls (like we had discussed, please let me know if that has changed).
Originally I had introduced the concept of Containers that could be serialized, but dropped that in favor of implementing the Serializable interface when needed instead.
@chingor13 I updated the AsyncReporter with the changes I applied to logging and pubsub as well, please take a look when you have a moment to let me know if you're happy with the direction.
It is now possible to flush any items from the queue prior to register_shutdown_function being triggered. This is intended mainly for the in memory queue and the primary use cases for this will be users who are writing their code inside of an event loop.

tmatsuo

The code LGTM

Let me conduct some perf test today

Sign in to view

-        return getenv('IS_BATCH_DAEMON_RUNNING') !== false;
+        $isDaemonRunning = filter_var(
+            getenv('IS_BATCH_DAEMON_RUNNING'),
+            FILTER_VALIDATE_BOOLEAN


chingor13 · 2017-06-27T18:16:46Z

The Trace AsyncReporter is working well in my test app.

michaelbausor · 2017-06-27T18:39:59Z

I have done a pass, LGTM!

tmatsuo · 2017-06-27T19:48:59Z

The performance of the e2e app is noticeably worse than the previous version. It's noticeable when you conduct a simple ab load test. The requests/second dropped from 269.92 to 134.32.

I think it is because the app handler needs to instantiate the LoggingClient which means it instantiate the internal Logger, where actually both the LoggingClient and Logger is not needed for the frontend request if we use the daemon.

The load test with the new version:

$ ab -n 10000 -c 20 https://tmatsuo-test.appspot.com/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking tmatsuo-test.appspot.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        tmatsuo-test.appspot.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        51 bytes

Concurrency Level:      20
Time taken for tests:   74.451 seconds
Complete requests:      10000
Failed requests:        5406
   (Connect: 0, Receive: 0, Length: 5406, Exceptions: 0)
Non-2xx responses:      1
Total transferred:      12633200 bytes
HTML transferred:       512079 bytes
Requests per second:    134.32 [#/sec] (mean)
Time per request:       148.903 [ms] (mean)
Time per request:       7.445 [ms] (mean, across all concurrent requests)
Transfer rate:          165.71 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    8   3.1      8      83
Processing:    50  140  45.6    130     732
Waiting:       50  140  45.6    129     732
Total:         57  149  45.9    138     741

Percentage of the requests served within a certain time (ms)
  50%    138
  66%    157
  75%    170
  80%    180
  90%    207
  95%    233
  98%    269
  99%    297
 100%    741 (longest request)

The load test with the old version

$ ab -n 10000 -c 20 https://logging-batch-old-dot-tmatsuo-test.appspot.com/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking logging-batch-old-dot-tmatsuo-test.appspot.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        logging-batch-old-dot-tmatsuo-test.appspot.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        50 bytes

Concurrency Level:      20
Time taken for tests:   37.047 seconds
Complete requests:      10000
Failed requests:        9962
   (Connect: 0, Receive: 0, Length: 9962, Exceptions: 0)
Total transferred:      12718268 bytes
HTML transferred:       517093 bytes
Requests per second:    269.92 [#/sec] (mean)
Time per request:       74.095 [ms] (mean)
Time per request:       3.705 [ms] (mean, across all concurrent requests)
Transfer rate:          335.25 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    8   2.6      8      77
Processing:    44   65  37.2     60     717
Waiting:       44   65  37.2     59     717
Total:         49   74  37.7     68     726

Percentage of the requests served within a certain time (ms)
  50%     68
  66%     71
  75%     74
  80%     75
  90%     83
  95%    103
  98%    164
  99%    191
 100%    726 (longest request)

The code for the new version:

<?php

require_once __DIR__ . '/../vendor/autoload.php';

use Google\Cloud\Logging\LoggingClient;

$client = new LoggingClient(
    ['projectId' => getenv('GCLOUD_PROJECT')]
);

$batchLogger = $client->psrLogger(
    'app',
    ['debugOutput' => false,
     'batchEnabled' => true,
     'batchOptions' => ['workerNum' => 10]
    ]
);

$num = isset($_GET['num']) ? $_GET['num'] : 1;
$size = isset($_GET['size']) ? $_GET['size'] : 1;

$start = microtime(true);

$message = str_repeat('x', $size);
for ($i = 0; $i < $num; $i++) {
    $batchLogger->info($message);
}
$end = microtime(true);
printf(
    '%d logs sent in %f seconds, at %f LPS' . PHP_EOL,
    $num, $end - $start,
    $num / ($end - $start)
);

The code for the old version:

<?php

require_once __DIR__ . '/../vendor/autoload.php';

use Google\Cloud\Logging\PsrBatchLogger;

$batchLogger = new PsrBatchLogger(
    'app',
    ['debugOutput' => false,
     'batchOptions' => ['workerNum' => 10],
     'clientConfig' => ['projectId' => getenv('GCLOUD_PROJECT')]
    ]
);

$num = isset($_GET['num']) ? $_GET['num'] : 1;
$size = isset($_GET['size']) ? $_GET['size'] : 1;

$start = microtime(true);

$message = str_repeat('x', $size);
for ($i = 0; $i < $num; $i++) {
    $batchLogger->info($message);
}
$end = microtime(true);
printf(
    '%d logs sent in %f seconds, at %f LPS' . PHP_EOL,
    $num, $end - $start,
    $num / ($end - $start)
);

tmatsuo · 2017-06-27T21:16:23Z

One possible idea of preventing the performance drop might be to make LoggingClient and the PsrLogger Lazily create the connection.

Unfortunately, it seems to me that, that change needs to have backward incompatible change, as long as we merge PsrLogger and PsrBatchLogger.

Our options are:

Make them lazily create the connection so that it'll have better performance, but we'll need to bump the major version of google/cloud and google/cloud-logging.
Have PsrLogger and PsrBatchLogger separate for now. Merge them when we bump the major version later.
Embrace the performance drop.

I prefer the option 1 over others. I'd push back the option 3.

mdietz94 · 2017-06-27T21:52:36Z

Just to be clear, this is a performance drop for users not using a daemon correct? I agree with preferring option 1 and pushing back on option 3.

tmatsuo · 2017-06-27T21:55:42Z

@mdietz94 This is a performance drop for users using the daemon. I don't think it affects users not using the daemon. Also, the drop is just for the LoggingClient.

mdietz94 · 2017-06-27T21:58:54Z

Meaning this won't affect CP/S users? I would have expected a very large increase in CP/S throughput/QPS with this change since it was synchronous before.

tmatsuo · 2017-06-27T22:01:55Z

@mdietz94

CP/S might have the same trivial issue, where the PubSubClient creates the internal Connection object when unnecessary, especially when the user is using the daemon.

But, yeah as you said, the new PubSubClient will have much much better performance compared to the previous state for some cases, even without using the daemon.

dwsupplee · 2017-06-27T22:02:31Z

Ouch! Yeah I don't like option 3 at all either. I toyed around a little bit and was able to get average timings for instantiating the LoggingClient down from 0.003524065s to 0.001411096s without the need for a breaking change.

@tmatsuo Would you be able to edit these lines

to

    /**
     * @param array $config
     */
    public function __construct(array $config = [])
    {
        $this->config = $config + [
            'serviceDefinitionPath' => __DIR__ . '/ServiceDefinition/logging-v2.json',
            'componentVersion' => LoggingClient::VERSION
        ];
    }

and add the following

        if (!$this->requestWrapper) {
            $this->setRequestWrapper(new RequestWrapper($this->config));
        }

        if (!$this->requestBuilder) {
            $this->setRequestBuilder(new RequestBuilder(
                $this->config['serviceDefinitionPath'],
                self::BASE_URI
            ));
        }

here to see if that improves the numbers a bit in another run?

dwsupplee · 2017-06-27T22:08:58Z

@mdietz94

A user will be able to directly instantiate the BatchPublisher class and use that without needing the overhead of creating the PubSubClient (which is why we're seeing a performance drop here). Either way they choose, however, they're still going to be seeing massive performance gains.

tmatsuo · 2017-06-27T22:13:10Z

@dwsupplee Yeah, it's little bit better

$ ab -n 10000 -c 20 https://tmatsuo-test.appspot.com/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking tmatsuo-test.appspot.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        tmatsuo-test.appspot.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        50 bytes

Concurrency Level:      20
Time taken for tests:   66.856 seconds
Complete requests:      10000
Failed requests:        9816
   (Connect: 0, Receive: 0, Length: 9816, Exceptions: 0)
Total transferred:      12637531 bytes
HTML transferred:       516196 bytes
Requests per second:    149.57 [#/sec] (mean)
Time per request:       133.713 [ms] (mean)
Time per request:       6.686 [ms] (mean, across all concurrent requests)
Transfer rate:          184.59 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    9   3.0      9      78
Processing:    46  124 137.3     89    1080
Waiting:       46  124 137.2     89    1080
Total:         55  133 137.4     98    1089

Percentage of the requests served within a certain time (ms)
  50%     98
  66%    108
  75%    115
  80%    122
  90%    167
  95%    286
  98%    784
  99%    869
 100%   1089 (longest request)

dwsupplee · 2017-06-27T22:20:56Z

~~I'm getting just about the exact same performance instantiating a LoggingClient with the changes I proposed just a moment ago vs. commenting out everything in the LoggingClient constructor.~~ edit: Nevermind, I misread the numbers - there is a decent boost when commenting out the connections.

If we want the absolute optimal performance while using the daemon, I think we might need to go back to having the PsrBatchLogger be a separate class. Even though it isn't used in batch mode, the overhead of creating the Logger looks like our bottleneck.

As another note, we will want to make sure to document this for users as I would imagine most would think to use the clients to gain access to the batch objects.

mdietz94 · 2017-06-27T22:23:34Z

Okay great, SGTM, thanks.

tmatsuo · 2017-06-27T23:00:50Z

I would imagine most would think to use the clients to gain access to the batch objects.

Yeah, I don't disagree.

I took another look at the both code and noticed that, the PsrBatchLogger also creates the Logger object before sending the entry via the message queue. So I think I was wrong.

tmatsuo · 2017-06-27T23:13:41Z

I think the performance drop came from the indirection of the BatchTrait and getCallback() which might be considered as a tax to pay for applying this to other services...

tmatsuo · 2017-06-27T23:20:24Z

OK, LGTM!

tmatsuo

New thought on the send function

Sign in to view

+            : new BatchRunner();
+        $this->batchRunner->registerJob(
+            $this->identifier,
+            [$this, 'send'],


tmatsuo · 2017-06-28T18:55:18Z

I'm conducting some load test again with an app which enables error reporting feature. With the new version, I occasionally see 500 error with an error at the bottom. I didn't see this behavior with the previous verison.

Error

A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "NOTICE: PHP message: PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\GoogleException: No project ID was provided, and we were unable to detect a default project ID. in /app/vendor/google/cloud/src/Core/ClientTrait.php:169"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "Stack trace:"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "#0 /app/vendor/google/cloud/src/Core/ClientTrait.php(81): Google\Cloud\Logging\LoggingClient->detectProjectId(Array)"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "#1 /app/vendor/google/cloud/src/Logging/LoggingClient.php(130): Google\Cloud\Logging\LoggingClient->configureAuthentication(Array)"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "#2 /app/vendor/google/cloud/src/ErrorReporting/Bootstrap.php(27): Google\Cloud\Logging\LoggingClient->__construct()"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "#3 /app/vendor/google/cloud/src/ErrorReporting/prepend.php(68): Google\Cloud\ErrorReporting\Bootstrap::init(NULL)"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "#4 {main}"
 
A  [28-Jun-2017 18:45:27] WARNING: [pool app] child 757 said into stderr: "  thrown in /app/vendor/google/cloud/src/Core/ClientTrait.php on line 169"
 
A  2017/06/28 18:45:27 [error] 10#0: *147640 FastCGI sent in stderr: "PHP message: PHP Warning:  file_get_contents(http://169.254.169.254/computeMetadata/v1/project/project-id): failed to open stream: Connection refused in /app/vendor/google/cloud/src/Core/Compute/Metadata/Readers/StreamReader.php on line 62
 
A  PHP message: PHP Fatal error:  Uncaught Google\Cloud\Core\Exception\GoogleException: No project ID was provided, and we were unable to detect a default project ID. in /app/vendor/google/cloud/src/Core/ClientTrait.php:169
 
A  Stack trace:
 
A  #0 /app/vendor/google/cloud/src/Core/ClientTrait.php(81): Google\Cloud\Logging\LoggingClient->detectProjectId(Array)
 
A  #1 /app/vendor/google/cloud/src/Logging/LoggingClient.php(130): Google\Cloud\Logging\LoggingClient->configureAuthentication(Array)
 
A  #2 /app/vendor/google/cloud/src/ErrorReporting/Bootstrap.php(27): Google\Cloud\Logging\LoggingClient->__construct()
 
A  #3 /app/vendor/google/cloud/src/ErrorReporting/prepend.php(68): Google\Cloud\ErrorReporting\Bootstrap::init(NULL)

tmatsuo · 2017-06-28T19:05:19Z

Then now I understand why I'm getting worse performance.

I tested the app with the ErrorReporting Bootstrap enabled. ErrorReporting then create a LoggingClient without specifying the projectId. It will issue a request to the metadata server for getting the project id.

tmatsuo · 2017-06-28T19:17:45Z

Yes, I get little better result

$ ab -n 10000 -c 20 https://tmatsuo-test.appspot.com/
This is ApacheBench, Version 2.3 <$Revision: 1604373 $>
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Licensed to The Apache Software Foundation, http://www.apache.org/

Benchmarking tmatsuo-test.appspot.com (be patient)
Completed 1000 requests
Completed 2000 requests
Completed 3000 requests
Completed 4000 requests
Completed 5000 requests
Completed 6000 requests
Completed 7000 requests
Completed 8000 requests
Completed 9000 requests
Completed 10000 requests
Finished 10000 requests


Server Software:        nginx
Server Hostname:        tmatsuo-test.appspot.com
Server Port:            443
SSL/TLS Protocol:       TLSv1.2,ECDHE-RSA-AES128-GCM-SHA256,2048,128

Document Path:          /
Document Length:        51 bytes

Concurrency Level:      20
Time taken for tests:   65.239 seconds
Complete requests:      10000
Failed requests:        7371
   (Connect: 0, Receive: 0, Length: 7371, Exceptions: 0)
Total transferred:      12638241 bytes
HTML transferred:       516432 bytes
Requests per second:    153.28 [#/sec] (mean)
Time per request:       130.477 [ms] (mean)
Time per request:       6.524 [ms] (mean, across all concurrent requests)
Transfer rate:          189.18 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        4    9   3.0      9      85
Processing:    49  121  36.2    116     452
Waiting:       49  120  36.1    115     451
Total:         62  130  36.5    125     458

Percentage of the requests served within a certain time (ms)
  50%    125
  66%    139
  75%    148
  80%    154
  90%    175
  95%    191
  98%    216
  99%    245
 100%    458 (longest request)

with having a file named ErrorReportingBootstrap in the project root.

<?php

require __DIR__ . '/vendor/autoload.php';

use Google\Cloud\Logging\LoggingClient;

return (new LoggingClient(['projectId' => getenv('GCLOUD_PROJECT')]))
    ->psrLogger('app-error', [
        'batchEnabled' => true,
        'debugOutput' => true,
        'batchOptions' => [
            'workerNum' => 2
        ]
    ]);

Does it make sense to read this environment variable before accessing Metadata server? This must affect other Client instantiation too.

tmatsuo · 2017-06-28T20:04:42Z

https://docs.google.com/document/d/126NJTDNctB0zL8dwTo8-9indyWGUXpGkrXCKrHU4b-s/edit#heading=h.8t69iwn5s7dr

Actually it broke the symphony integration in the above doc.
Does it make sense to revive the PsrBatchLogger for easier dependency injection?

Edit:
Maybe we can just have a static factory method somewhere.

dwsupplee · 2017-06-28T20:53:07Z

Good find, that explains the differences between our test runs as I was running my code locally.

Does it make sense to read this environment variable before accessing Metadata server? This must affect other Client instantiation too.

Yes! That sounds like a great idea.

Actually it broke the symphony integration in the above doc.
Does it make sense to revive the PsrBatchLogger for easier dependency injection?

Would configuring a factory help solve this problem?

tmatsuo · 2017-06-28T21:08:42Z

Created #551

@see

* introduce batch publisher * Add copyright * batch updates * fix bootstrap * add coverage for batchPublisher * remove unused imports * fix @see link

@see

* introduce batch publisher * Add copyright * batch updates * fix bootstrap * add coverage for batchPublisher * remove unused imports * fix @see link

@see

* introduce batch publisher * Add copyright * batch updates * fix bootstrap * add coverage for batchPublisher * remove unused imports * fix @see link

@see

* Adding Batch package and PsrBatchLogger. * Added appengine.googleapis.com/trace_id for log request correlation. * Evaluate the $_SERVER for each request for creating labels for log request correlation. * Correctly add the labels * Added a test around the TraceId * Fixed wrong tests * Evaluate $server in the constructor. * Added metadataProvider to PsrLogger * Fix the document generation error (#469) * Dynamically change the number of children (#489) * Dynamically change the number of children * Add AsyncReporter for trace which relies on the batch runner (#548) * Introduce batch publisher (#534) * introduce batch publisher * Add copyright * batch updates * fix bootstrap * add coverage for batchPublisher * remove unused imports * fix @see link * Batch branch update (#551) * Documentation update * Use GCLOUD_PROJECT envvar for project id detection. * Added a static factory method for creating batch enabled PsrLogger * Force enabling the batch mode

dwsupplee added the api: pubsub Issues related to the Pub/Sub API. label Jun 13, 2017

googlebot added the cla: yes This human has signed the Contributor License Agreement. label Jun 13, 2017

tmatsuo reviewed Jun 14, 2017

View reviewed changes

Comment thread src/PubSub/BatchPublishContainer.php Outdated

}

$end = microtime(true);

if ($this->debugOutput) {

printf(

This comment was marked as spam.

Sign in to view

dwsupplee added 3 commits June 26, 2017 20:54

introduce batch publisher

6f404b7

Add copyright

b15a1bd

batch updates

ada2dab

dwsupplee force-pushed the batch-publisher branch from 8b6fdf8 to ada2dab Compare June 27, 2017 17:12

dwsupplee added 3 commits June 27, 2017 13:35

fix bootstrap

969a2fa

add coverage for batchPublisher

7d344c2

remove unused imports

d03b6a1

tmatsuo reviewed Jun 27, 2017

View reviewed changes

Comment thread src/Core/Batch/SysvTrait.php

return getenv('IS_BATCH_DAEMON_RUNNING') !== false;

$isDaemonRunning = filter_var(

getenv('IS_BATCH_DAEMON_RUNNING'),

FILTER_VALIDATE_BOOLEAN

This comment was marked as spam.

Sign in to view

fix @see link

5a9c89d

tmatsuo approved these changes Jun 27, 2017

View reviewed changes

tmatsuo reviewed Jun 28, 2017

View reviewed changes

dwsupplee merged commit 2c7b765 into googleapis:batch Jun 28, 2017

This was referenced Jun 30, 2017

Introduce Batching #556

Merged

Add asynchronous publish support in Pubsub #458

Closed

jdpedrie mentioned this pull request Jun 30, 2017

Prepare v0.33.0 #557

Merged

Uh oh!

Conversation

dwsupplee commented Jun 13, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example Usage

Notes on the implementation

Caveats

Uh oh!

tmatsuo commented Jun 14, 2017

Uh oh!

This comment was marked as spam.

Uh oh!

dwsupplee commented Jun 19, 2017

Uh oh!

mdietz94 commented Jun 19, 2017 via email

Uh oh!

dwsupplee commented Jun 27, 2017

Uh oh!

tmatsuo left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

chingor13 commented Jun 27, 2017

Uh oh!

michaelbausor commented Jun 27, 2017

Uh oh!

tmatsuo commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

mdietz94 commented Jun 27, 2017 via email

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

mdietz94 commented Jun 27, 2017 via email

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

dwsupplee commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dwsupplee commented Jun 27, 2017

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

dwsupplee commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mdietz94 commented Jun 27, 2017 via email

Uh oh!

tmatsuo commented Jun 27, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

tmatsuo commented Jun 27, 2017

Uh oh!

tmatsuo left a comment

Choose a reason for hiding this comment

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

This comment was marked as spam.

Uh oh!

tmatsuo commented Jun 28, 2017

Uh oh!

tmatsuo commented Jun 28, 2017

Uh oh!

tmatsuo commented Jun 28, 2017

Uh oh!

tmatsuo commented Jun 28, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

dwsupplee commented Jun 28, 2017

dwsupplee commented Jun 13, 2017 •

edited

Loading

tmatsuo commented Jun 27, 2017 •

edited

Loading

dwsupplee commented Jun 27, 2017 •

edited

Loading

dwsupplee commented Jun 27, 2017 •

edited

Loading

tmatsuo commented Jun 27, 2017 •

edited

Loading

tmatsuo commented Jun 28, 2017 •

edited

Loading