running 2 instances of scancode hangs

[55jr55]

When I run two instances of scancode, each against a different source tree with no overlap, both instances hang almost immediately. Are there any options available for debugging?

[pombredanne]

@55jr55 Which version of ScanCode are you running exactly? Which command line options are you using and on which operating system?

[55jr55]

Sorry, should have included that.

Scancode 2.0
Linux debian 3.2.0-4-486 #1 Debian 3.2.63-2+deb7u1 i686 GNU/Linux
command line: ~/scancode/sc-2.0/scancode -f json -l /home/user/intel_usg/atom ~/macShare/sc-atom.json

[pombredanne]

@55jr55 Thanks. When you say 2.0, I assume you meant the 2.0.0rc1 release candidate and not a develop branch checkout, right?

[pombredanne]

Hum... I was not able to reproduce this.
Here are the two possibilities where multi-processes could step on each other toes:

1). On a fresh checkout or download, when you run scancode for the first time across any process, it self-configures itself meaning it creates a Python virtualenv isolated environment and installs all the vendored third-party libraries. This is something that happens only once and there NO mechanism to guard this with a file-based exclusive lock to ensure that only one process can execute this.

Was this a fresh extract on first run?

2). When you run a license detection for the first time on a given installation, an index of license texts and rules is built. This is guarded by a multi-processes-safe and thread-safe file-based exclusive lock mechanism and ensure that the first process that acquires the lock creates the index and caches on disk while any other process waits for that to complete and then loads the cached index. This is fairly heavily tested so I would doubt this would be the place where this could hang.

If you had run scancode license detection at least one before, then the cache would be have created and processes would just acquire a lock, load the index and release the lock fairly quickly.

So I guess this is because of 1. and not of 2.

Could you run this for me?
./configure --clean
./scancode -h

Then once this is completed, start again your two scans in parallel and report if there is still a problem?

[55jr55]

I'd be happy to re-run after doing the clean and -h. I'm confused about which version of 2.0 I should be using. I had downloaded the zipped version of 2.0 about a week ago. I assume that was the develop branch. I've also cloned the repository so I can checkout and run from there. I don't see any release candidate though.
user@debian:~/scancode/sc-2.0-git/scancode-toolkit$ git branch -a
-develop
remotes/origin/111-new-version-notification
remotes/origin/143-incorrect-license-detection
remotes/origin/212-chart-summary-display
remotes/origin/214-license-policy
remotes/origin/219-multiprocessing
remotes/origin/253-python-packages
remotes/origin/262-shallow-extraction
remotes/origin/300-extract-weird-name-on-windows
remotes/origin/320-links-open-in-new-tab
remotes/origin/327-license-detection-accuracy-improvements
remotes/origin/HEAD -> origin/develop
remotes/origin/develop
remotes/origin/master
remotes/origin/orm-models

[pombredanne]

ok, so you are on the develop branch. This should work as well as you as long as you pull the latest AND run ./configure --clean (which you should rerun each time you make a pull when working from a Git clone in case new or updated third-parties were pushed since your last pull)

Note that we typically do not create branches for releases in most cases, only a tag (and stable release are in master, unstable are in develop)
FWIW, the 2.0.0rc1 is at the top of the release page: https://github.com/nexB/scancode-toolkit/releases

[55jr55]

I ran the two concurrent scans with 2.0rc1 and it does not hang. However, one of the scans dies with this error:

[----------------#-------------------] 37166 Scanning: CSceAppInitializer.cp
[----------------#-------------------] 37167 Scanning: CMixGenerated.h /home/user/scancode/sc-2.0rc1/scancode: line 16: 6371 Killed $SCANCODE_ROOT_DIR/bin/scancode "$@"

It does this with 2.0 dev and rc1 and each time it dies on file 37167.

[pombredanne]

ok, then this a different issue with I think is the same as explained here: #267 (comment)
.... One process gets proc-killed by the Linux Kernel, likely because for now we still accumulate all results in memory rather than streaming them to disk. We are working on this as explained in #267 (comment)

The short-term work around is to try to scan smaller directory trees at once (or get much more RAM which is not practical of course). And eventually to avoid running multiple large scans in parallel.

I will keep posted as soon as we have a the fix to stream things to disk

[pombredanne]

The branch cache-and-stream-scan-results in #354 should have a reasonable working solution that caches scan results on disk and then stream reads the cached results to create the final JSON output at the end. This should avoid pathological cases of large scans with million files being held in memory and eventually exhausting RAM and should fix your issue.

The fix was developed for this and #267

[pombredanne]

At this stage your bug should be fixed in develop. Closing this now.