deduplicate warnings across multi-node runs

[rohany]

Currently, warnings that occur are duplicated when running on multiple nodes. This is a bit annoying as the legate driver handles deduplication of stdout.

[manopapad]

What warnings are you seeing duplicated?

[rohany]

For example, anything that comes out of warnings.warn -- things like "this function is not yet implemented by cuNumeric" etc.

[manopapad]

This is happening because warnings.warn prints out to stderr, which is not deduplicated like stdout https://github.com/StanfordLegion/legion/blob/control_replication/bindings/python/legion_top.py#L112.

I can see (at least) 3 ways around this:

• Deduplicate stderr in legion_top.py in the same way that we do for stdout. I would personally rather be conservative and avoid hiding stderr, just in case something goes wrong on one shard specifically.
• Hide warnings.warn on all shards except 0, globally in legion_top.py.
• Do the same but in legate/cunumeric.

@elliottslaughter are you interested in doing this globally for all python execution within Legion?

[rohany]

I also agree that deduplicating stderr is not a good idea due to shard local errors.

Hide warnings.warn on all shards except 0, globally in legion_top.py.

Is this something we want to do globally, or instead have legate provide a separate warnings module that handles it?

[manopapad]

That's another option, use a custom warnings module specifically in cunumeric, that takes care of this internally (or just prints to stdout). Or we just ask the warnings module to use stdout inside of legate.

I just wanted to see if @elliottslaughter wanted to handle this globally in Legion, since warnings is a standard library module.

[elliottslaughter]

Are you proposing monkey-patching the warnings module? Or is there a nicer way to do this without doing that?

The monkey-patching approach seems a bit scary to me, to be honest. Unless the warnings module is specifically designed with an integration point for this, it seems risky.

I'd be fine with a legion_warnings module (or maybe legion_top.warnings; not sure). Point being to make an API-compatible wrapper that does what you need. It can be centralized, so that we don't have N implementations. And it avoids the need for monkey-patching. That doesn't handle other non-Legion-aware libraries, which is a drawback. But Legion-aware libraries can opt in to it.

[manopapad]

Are you proposing monkey-patching the warnings module? Or is there a nicer way to do this without doing that?

We can globally set the warnings module to print to stdout instead of stderr.

From https://docs.python.org/3/library/warnings.html#warnings.showwarning:

warnings.showwarning(message, category, filename, lineno, file=None, line=None)
Write a warning to a file. The default implementation calls formatwarning(message, category, filename, lineno, line) and writes the resulting string to file, which defaults to sys.stderr. You may replace this function with any callable by assigning to warnings.showwarning.

From https://stackoverflow.com/a/858928:

def customwarn(message, category, filename, lineno, file=None, line=None):
    sys.stdout.write(warnings.formatwarning(message, category, filename, lineno))

warnings.showwarning = customwarn

[elliottslaughter]

Yup, if it's explicitly designed to be used like this then I think it's fair game to set warnings.showwarning in legion_top.py.

[manopapad]

This was fixed with https://gitlab.com/StanfordLegion/legion/-/merge_requests/618, thanks @rohany !