Name and publisher normalization for cross reference

[JohnMcPMS]

Change

This change adds code for creating normalized strings from package names and publishers. It is not actually used anywhere yet, but it is intended for use in correlating locally installed packages with remote ones, as well correlating one version to the next. I also plan to use the values to create "nicer" Ids for locally installed packages that are not found in a remote source.

In order to support all of Unicode, the code does not use STL's regex, but instead an ICU regex wrapper is implemented to enable global support. This wrapper's interface is purpose built for this use case, so it may need to be adapted as needed for other uses.

The expressions and their use were adapted from code that is used to do similar work for app compatibility on Windows. The goal is to remove versions and extraneous strings from names and publishers to create values unique to the package. The code also attempts to capture the architecture and locale if it is present in the name.

A large set of real world examples can be found in the test data files. These contain real name InputNames.txt and publisher InputPublishers.txt strings from packages found in winget-pkgs, as well as a few others added to smoke test proper functionality outside of English and other European character sets. The result file NormalizationInitialIds.txt contains the strings that result from normalizing both name and publisher and appending them as Publisher + . + Name.

Microsoft Reviewers: Open in CodeFlow

[jsoref]

I'd generally suggest doing:

Suggested change

	\b\p{Lu}{2,3}(-(CANS|CYRL|LATN|MONG))?-\p{Lu}{2}(?![A-Z])(?:-VALENCIA)?\b
	\b\p{Lu}{2,3}(?:-(?:CANS|CYRL|LATN|MONG))?-\p{Lu}{2}(?![A-Z])(?:-VALENCIA)?\b

Basically, if you aren't capturing you should tell perl you aren't.

Otherwise, if you have a rule like:

\b([A-Za-z])\1{3,}\b

(which is earlier in this file), you may be unhappy.

[JohnMcPMS]

The entire file is run as one big regex? I would assume that the capture reference would be per-line. I don't mind changing it, I just didn't expect that the captures really mattered for this use case.

[jsoref]

Yeah, it's run as one big regex.

It's a lazy optimization that only trips things up occasionally.

I need to look into whether it's worth treating each line as a unique regex / how to make that work.

[yao-msft]

L"ZH-CN" [](start = 86, length = 8)

The comment said the values must be in sorted order but looks like no?

[JohnMcPMS]

They also need to be folded, which is a harder thing to expect people to do. So I fold and sort them into another container in the constructor. I can update the comment to better reflect this.

[yao-msft]

@@ [](start = 45, length = 2)

what's special with this?

[JohnMcPMS]

I have no idea; the starting point for this code was something doing similar work, and I left the majority of it alone.

[yao-msft]

[github-actions]

Misspellings found, please review:

• URIs

To accept these changes, run the following commands from this repository on this branch

pushd $(git rev-parse --show-toplevel)
perl -e '
my $new_expect_file=".github/actions/spelling/expect.txt";
use File::Path qw(make_path);
make_path ".github/actions/spelling";
open FILE, q{<}, $new_expect_file; chomp(my @words = <FILE>); close FILE;
my @add=qw('"URIs "');
my %items; @items{@words} = @words x (1); @items{@add} = @add x (1);
@words = sort {lc($a) cmp lc($b)} keys %items;
open FILE, q{>}, $new_expect_file; for my $word (@words) { print FILE "$word\n" if $word =~ /\w/; };
close FILE;'
popd