Performance: only match first character for gem suggestions#20
Closed
gstark wants to merge 1 commit into
Closed
Conversation
…name matches Comparing all gem names via levenshtein_distance is slow enough to introduce a long pause before printing gem suggestions
Member
|
Again. No. I've explained why multiple times now. |
junaruga
added a commit
to junaruga/rubygems
that referenced
this pull request
Jun 9, 2026
When `bundle install` connects with a certification (CA) to a private RubyGems
HTTPS server emulated by WEBrick, the connection to the
https://localhost:18443/versions succeeded, but the connection to download gems
failed with the following error.
```
+ /home/jaruga/var/git/ruby/rubygems/bin/bundle config set --local ssl_ca_cert /home/jaruga/git/report-bundler-bundle-config-set-ssl_ca_cert/tmp/client/ssl/ca.crt
...
+ /home/jaruga/var/git/ruby/rubygems/bin/bundle install -V
Running `bundle install --verbose` with bundler 4.1.0.dev
Resolving dependencies because there's no lockfile
HTTP GET https://localhost:18443/versions
HTTP 206 Partial Content https://localhost:18443/versions
HTTP GET https://localhost:18443/versions
HTTP 200 OK https://localhost:18443/versions
Fetching gem metadata from https://localhost:18443/
Looking up gems ["hello"]
Resolving dependencies...
Using bundler 4.1.0.dev
1: bundler (4.1.0.dev) from /home/jaruga/var/git/ruby/rubygems/bundler/bundler.gemspec
Fetching hello 0.1.0
Retrying download gem from https://localhost:18443/ due to error (2/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 1.22 seconds before retry
Retrying download gem from https://localhost:18443/ due to error (3/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 2.26 seconds before retry
Retrying download gem from https://localhost:18443/ due to error (4/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 4.02 seconds before retry
Bundler::InstallError: Bundler::HTTPError: Could not download gem from https://localhost:18443/ due to underlying error <SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)>
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/rubygems_integration.rb:406:in 'Bundler::RubygemsIntegration#download_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:531:in 'block in Bundler::Source::Rubygems#download_gem'
/home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems.rb:1068:in 'Gem.time'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:530:in 'Bundler::Source::Rubygems#download_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:459:in 'Bundler::Source::Rubygems#fetch_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:443:in 'Bundler::Source::Rubygems#fetch_gem_if_possible'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:575:in 'Bundler::Source::Rubygems#rubygems_gem_installer'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:184:in 'Bundler::Source::Rubygems#download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/gem_installer.rb:29:in 'Bundler::GemInstaller#download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/parallel_installer.rb:148:in 'Bundler::ParallelInstaller#do_download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/parallel_installer.rb:132:in 'block in Bundler::ParallelInstaller#worker_pool'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:70:in 'Bundler::Worker#apply_func'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:65:in 'block in Bundler::Worker#process_queue'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:56:in 'Kernel#loop'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:56:in 'Bundler::Worker#process_queue'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:98:in 'block (2 levels) in Bundler::Worker#create_threads'
...
```
`Bundler::Fetcher` has the `#connection` calling `#bundler_cert_store` storing
`Bundler.settings[:ssl_ca_cert]` in the following part. It is used in some parts.
bundler/lib/bundler/fetcher.rb
...
def bundler_cert_store
...
ssl_ca_cert = Bundler.settings[:ssl_ca_cert] ||
(Gem.configuration.ssl_ca_cert if
Gem.configuration.respond_to?(:ssl_ca_cert))
...
end
```
However in the case of downloading gems in Bundler,
Bundler calls `Bundler::Source::Rubygems#download_gem`
calling `Bundler::Fetcher#gem_remote_fetcher`
calling `Bundler::Fetcher::GemRemoteFetcher`
extending `Gem::RemoteFetcher` managing `@cert_files` for RubyGems.
Therefore, the `Bundler::Fetcher::GemRemoteFetcher` needs to update the `@cert_files`
by adding the value of the `Bundler.settings[:ssl_ca_cert]`.
As `Gem::Request.configure_connection_for_https` is called, and gets the value of
`Gem.configuration.ssl_ca_cert` in the proess of downloading gems,
we don't need to add the value in the `Bundler::Fetcher::GemRemoteFetcher`.
The backtrace is below.
```
+ rdbg -c \
-e 'b Gem::Request.configure_connection_for_https' \
-e c \
-- /home/jaruga/git/ruby/rubygems/bin/bundle install -V
...
(rdbg) bt # backtrace command
=>#0 Gem::Request.configure_connection_for_https(connection=#<Gem::Net::HTTP localhost:18443 open=fal..., cert_files=["/home/jaruga/.local/ruby-4.1.0-debug-3e...) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request.rb:71
ruby#1 Gem::Request::HTTPSPool#setup_connection(connection=#<Gem::Net::HTTP localhost:18443 open=fal...) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request/https_pool.rb:7
ruby#2 Gem::Request::HTTPPool#make_connection at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request/http_pool.rb:43
ruby#3 Gem::Request::HTTPPool#checkout at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request/http_pool.rb:23
ruby#4 Gem::Request#connection_for(uri=#<Gem::Uri:0x00007f4e473adda0 @parsed_uri...) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request.rb:136
ruby#5 Gem::Request#perform_request(request=#<Gem::Net::HTTP::Get GET path="/gems/hel...) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request.rb:194
ruby#6 Gem::Request#fetch at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request.rb:161
ruby#7 Gem::RemoteFetcher#request(uri=#<Gem::Uri:0x00007f4e473adda0 @parsed_uri..., request_class=Gem::Net::HTTP::Get, last_modified=nil) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/remote_fetcher.rb:326
ruby#8 Bundler::Fetcher::GemRemoteFetcher#request(args=[#<Gem::Uri:0x00007f4e473adda0 @parsed_ur...) at ~/var/git/ruby/rubygems/bundler/lib/bundler/fetcher/gem_remote_fetcher.rb:18
ruby#9 Gem::RemoteFetcher#fetch_http(uri=#<Gem::Uri:0x00007f4e473adda0 @parsed_uri..., last_modified=nil, head=false, depth=0) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/remote_fetcher.rb:217
ruby#10 Gem::RemoteFetcher#fetch_path(uri=#<Gem::Uri:0x00007f4e473adda0 @parsed_uri..., mtime=nil, head=false) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/remote_fetcher.rb:261
ruby#11 Gem::RemoteFetcher#cache_update_path(uri=#<Gem::URI::HTTPS https://localhost:18443..., path="/home/jaruga/var/git/report-bundler-bund..., update=true) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/remote_fetcher.rb:302
ruby#12 block in Bundler::RubygemsIntegration#download_gem (2 levels) at ~/var/git/ruby/rubygems/bundler/lib/bundler/rubygems_integration.rb:393
ruby#13 Bundler::SharedHelpers#filesystem_access(path="/home/jaruga/var/git/report-bundler-bund..., action=:write, block=#<Proc:0x00007f4e1d7dfad8 /home/jaruga/va...) at ~/var/git/ruby/rubygems/bundler/lib/bundler/shared_helpers.rb:106
ruby#14 block in Bundler::RubygemsIntegration#download_gem at ~/var/git/ruby/rubygems/bundler/lib/bundler/rubygems_integration.rb:392
ruby#15 Bundler::Retry#run(block=#<Proc:0x00007f4e1d7df9c0 /home/jaruga/va...) at ~/var/git/ruby/rubygems/bundler/lib/bundler/retry.rb:48
ruby#16 Bundler::Retry#attempts(block=#<Proc:0x00007f4e1d7df880 /home/jaruga/va...) at ~/var/git/ruby/rubygems/bundler/lib/bundler/retry.rb:38
ruby#17 Bundler::RubygemsIntegration#download_gem(spec=Gem::Specification.new do |s| s.name = "h..., uri=#<Gem::URI::HTTPS https://localhost:18443..., cache_dir="/home/jaruga/var/git/report-bundler-bund..., fetcher=#<Bundler::Fetcher::GemRemoteFetcher:0x00...) at ~/var/git/ruby/rubygems/bundler/lib/bundler/rubygems_integration.rb:384
ruby#18 block in Bundler::Source::Rubygems#download_gem at ~/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:531
ruby#19 Gem.time(msg="Downloaded hello in", width=0, display=true) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems.rb:1068
ruby#20 Bundler::Source::Rubygems#download_gem(spec=Gem::Specification.new do |s| s.name = "h..., download_cache_path="/home/jaruga/var/git/report-bundler-bund..., previous_spec=nil) at ~/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:530
...
```
According to the following logic, `@cert_files` is not nil,
as `Dir.glob(patterns` returns Array.
lib/rubygems/remote_fetcher.rb
```
def initialize(proxy = nil, dns = nil, headers = {})
...
@cert_files = Gem::Request.get_cert_files
...
end
```
lib/rubygems/request.rb
```
def self.get_cert_files
pattern = File.expand_path("./ssl_certs/*/*.pem", __dir__)
Dir.glob(pattern)
end
```
junaruga
added a commit
to junaruga/rubygems
that referenced
this pull request
Jun 9, 2026
When `bundle install` connects with a certification (CA) to a private RubyGems
HTTPS server emulated by WEBrick, the connection to the
https://localhost:18443/versions succeeded, but the connection to download gems
failed with the following error.
```
+ /home/jaruga/var/git/ruby/rubygems/bin/bundle config set --local ssl_ca_cert /home/jaruga/git/report-bundler-bundle-config-set-ssl_ca_cert/tmp/client/ssl/ca.crt
...
+ /home/jaruga/var/git/ruby/rubygems/bin/bundle install -V
Running `bundle install --verbose` with bundler 4.1.0.dev
Resolving dependencies because there's no lockfile
HTTP GET https://localhost:18443/versions
HTTP 206 Partial Content https://localhost:18443/versions
HTTP GET https://localhost:18443/versions
HTTP 200 OK https://localhost:18443/versions
Fetching gem metadata from https://localhost:18443/
Looking up gems ["hello"]
Resolving dependencies...
Using bundler 4.1.0.dev
1: bundler (4.1.0.dev) from /home/jaruga/var/git/ruby/rubygems/bundler/bundler.gemspec
Fetching hello 0.1.0
Retrying download gem from https://localhost:18443/ due to error (2/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 1.22 seconds before retry
Retrying download gem from https://localhost:18443/ due to error (3/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 2.26 seconds before retry
Retrying download gem from https://localhost:18443/ due to error (4/4): Gem::RemoteFetcher::FetchError SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)
Sleeping for 4.02 seconds before retry
Bundler::InstallError: Bundler::HTTPError: Could not download gem from https://localhost:18443/ due to underlying error <SSL_connect returned=1 errno=0 peeraddr=127.0.0.1:18443 state=error: certificate verify failed (unable to get local issuer certificate) (https://localhost:18443/gems/hello-0.1.0.gem)>
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/rubygems_integration.rb:406:in 'Bundler::RubygemsIntegration#download_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:531:in 'block in Bundler::Source::Rubygems#download_gem'
/home/jaruga/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems.rb:1068:in 'Gem.time'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:530:in 'Bundler::Source::Rubygems#download_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:459:in 'Bundler::Source::Rubygems#fetch_gem'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:443:in 'Bundler::Source::Rubygems#fetch_gem_if_possible'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:575:in 'Bundler::Source::Rubygems#rubygems_gem_installer'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:184:in 'Bundler::Source::Rubygems#download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/gem_installer.rb:29:in 'Bundler::GemInstaller#download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/parallel_installer.rb:148:in 'Bundler::ParallelInstaller#do_download'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/installer/parallel_installer.rb:132:in 'block in Bundler::ParallelInstaller#worker_pool'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:70:in 'Bundler::Worker#apply_func'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:65:in 'block in Bundler::Worker#process_queue'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:56:in 'Kernel#loop'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:56:in 'Bundler::Worker#process_queue'
/home/jaruga/var/git/ruby/rubygems/bundler/lib/bundler/worker.rb:98:in 'block (2 levels) in Bundler::Worker#create_threads'
...
```
`Bundler::Fetcher` has the `#connection` calling `#bundler_cert_store` storing
`Bundler.settings[:ssl_ca_cert]` in the following part. It is used in some parts.
bundler/lib/bundler/fetcher.rb
...
def bundler_cert_store
...
ssl_ca_cert = Bundler.settings[:ssl_ca_cert] ||
(Gem.configuration.ssl_ca_cert if
Gem.configuration.respond_to?(:ssl_ca_cert))
...
end
```
However in the case of downloading gems in Bundler,
Bundler calls `Bundler::Source::Rubygems#download_gem`
calling `Bundler::Fetcher#gem_remote_fetcher`
calling `Bundler::Fetcher::GemRemoteFetcher`
extending `Gem::RemoteFetcher` managing `@cert_files` for RubyGems.
Therefore, the `Bundler::Fetcher::GemRemoteFetcher` needs to update the `@cert_files`
by adding the value of the `Bundler.settings[:ssl_ca_cert]`.
As `Gem::Request.configure_connection_for_https` is called, and gets the value of
`Gem.configuration.ssl_ca_cert` in the proess of downloading gems,
we don't need to add the value in the `Bundler::Fetcher::GemRemoteFetcher`.
The backtrace is below.
```
+ rdbg -c \
-e 'b Gem::Request.configure_connection_for_https' \
-e c \
-- /home/jaruga/git/ruby/rubygems/bin/bundle install -V
...
(rdbg) bt # backtrace command
=>#0 Gem::Request.configure_connection_for_https(connection=#<Gem::Net::HTTP localhost:18443 open=fal..., cert_files=["/home/jaruga/.local/ruby-4.1.0-debug-3e...) at ~/.local/ruby-4.1.0-debug-3ef48ef9c8-openssl-4.1.0-7194354488/lib/ruby/4.1.0+1/rubygems/request.rb:71
...
ruby#20 Bundler::Source::Rubygems#download_gem(spec=Gem::Specification.new do |s| s.name = "h..., download_cache_path="/home/jaruga/var/git/report-bundler-bund..., previous_spec=nil) at ~/var/git/ruby/rubygems/bundler/lib/bundler/source/rubygems.rb:530
...
```
According to the following logic, `@cert_files` is not nil,
as `Dir.glob(patterns` returns Array.
lib/rubygems/remote_fetcher.rb
```
def initialize(proxy = nil, dns = nil, headers = {})
...
@cert_files = Gem::Request.get_cert_files
...
end
```
lib/rubygems/request.rb
```
def self.get_cert_files
pattern = File.expand_path("./ssl_certs/*/*.pem", __dir__)
Dir.glob(pattern)
end
```
This pull request was closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First, thanks for accepting my patch. I like the changes made to it. However, the performance constraint which only considered gems that matched the first character didn't survive the change.
I originally introduced this because there is a noticeable delay generating the list of suggestions. Matching the first character seemed like a reasonable constraint as I figured most mistakes will not be in the first character.
Perhaps gem users won't dislike the lookup time, or perhaps there is a different idea on how to make this code faster.
Happy New Year, and thanks again for accepting my original patch.