Windows page fault disk i/o slow on first load

Hello,

As of #613 I have experienced significant regression in model loading speed (I'm on windows, compiled msvc llama.cpp, llama.cpp is located on HDD to prevent SSD wear in my case)

It takes roughly 15 minutes for model to load first time after each computer restart/hibernation, during this time my HDD usage is at 100% and my non-llama.cpp read/write operations are slowed down on my pc

Before that, previous commits took 60 - 180 seconds at worst to load model first time, and after first loading occured, model loaded within 5 - 10 seconds on each program restart until pc reboot/hibernation

Before Commit:

After:

I see reason why model might load faster for some while slower (like my case) for others after recent changes, therefore in my opinion best solution is adding parameter that lets people disable llama.cpp's recent model loading changes if thats possible

Thanks

I'm sorry page fault disk i/o on Windows is going slow for you!

The best we can do here is find some trick to coax better performance out of your Windows executive. Could you please try something for me? Take this code:

        HANDLE proc;
        WIN32_MEMORY_RANGE_ENTRY entry;
        proc = GetCurrentProcess();
        entry.VirtualAddress = addr;
        entry.NumberOfBytes = length;
        if (!PrefetchVirtualMemory(proc, 1, &entry, 0)) abort();

And put it at the end of the mmap_file() function in llama.cpp. Then compile. Reboot, Try again. If it helps, we'll use that.

Do you have a spinning disk? If you do, then another thing we could try if prefetch doesn't work, is having a loop where we do a volatile poke of each memory page so they're faulting sequentially. That could help the executive to dispatch i/o in a way that minimizes disk head movement. Although it's more of a long shot.

Other than those two things, I'm not sure how we'd help your first case load times. The good news though is that subsequent runs of the LLaMA process will go much faster than they did before!

There is still time to revert all this mess and go back to the previous model format.

I'm not saying mmapping couldn't or shouldn't be implemented, but that there is a right way and a wrong way and the current way is simply wrong. Unnecessarily padding the model files with overhead what amounts to basically "anti-compression" or "bloat" and making them incompatible when there are proper ways to achieve this like with the new kv_cache storing/loading mechanism implemented in #685 which wouldn't introduce any regressions.

I got similar issue as you, although on linux. Loading speed is unsatifying with current approach.

@anzz1

@anzz1 100% agree with you there

I think FILE_MAP_LARGE_PAGES should be used as a dwDesiredAccess in MapViewOfFileEx and SEC_LARGE_PAGES on flProtect in CreateFileMapping because I think 1000 megabyte requests are better than 1000000 kilobyte requests.

I don't know if it helps, but there is also a mmap64 function for loading larger files.

FILE_MAP_LARGE_PAGES requires special privileges. I've never managed to figure out how to configure Windows to let me use it at all. As far as I know, it's mostly only useful for optimizing TLB access. Very little chance it'd help us here.

Read https://learn.microsoft.com/en-us/windows/win32/memory/creating-a-file-mapping-using-large-pages
This will help you with privileges.

Allocating huge pages using mmap

You can allocate huge pages using mmap(...MAP_HUGETLB). Either as a anonymous mapping or a named mapping on hugetlbfs. The size of a huge page mapping needs to be a multiple of the page size. For example to create an anonymous mapping of 8 huge pages of the default size of 2 MiB on AMD64 / x86-64:

void *ptr = mmap(NULL, 8 * (1 << 21), PROT_READ | PROT_WRITE,
MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB,
-1, 0))
https://rigtorp.se/hugepages/

@x02Sylvie

@x02Sylvie I noticed that the new conversion tool creates files with a lot of fragments (in my case I measured 3m fragments in the 65B model). This is not much of a problem with an SSD, but with an spinning disk it may increase load times dramatically. Can you try defragmenting the disk or model file?

You can allocate huge pages using mmap(...MAP_HUGETLB).

Linux requires that you decide ahead of time what percent of memory is going to be devoted to huge pages and nothing else. With LLaMA that means you'd have to give up the lion's share of your memory, just so that when you run the llama command on occasion it goes maybe 10% faster? A:t best? I haven't measured it, because the last time I tried doing this, I totally nuked my system. I imagine this huge page thing might be nice if you've got something like a Postgres database server where it can have a more clearly defined subset of memory that's your work area. But I'm pretty doubtful that, given Linux's huge TLB design, and its requirement of special privileges and systems configuration, that it's really something worth focusing on.

3 gb/s without mmap read compared to 500 mb/s read with mmap

3 gb/s without mmap read compared to 500 mb/s read with mmap

With how many threads?

SSDs, you know, also prefer large linear reads.

I tried with 1 and 8 threads same result 500 mb/s read

SSDs, you know, also prefer large linear reads.

NVMe was invented because SSDs prefer parallel IO

There is also a good way, first map the file then create multiple async read requests to one mb allocated memory in ram.
Or just another thread that reads the file

Solved first model loading time in #734

Assuming that the model fits in RAM, and the guess about the threads jumping over each other is correct, the workaround would be to make sure that the first pass over the weights happens linearly.

Better to use an explicit hint: PrefetchVirtualMemory on Windows and madvise on Unix.

Though I suspect this is unnecessary if you use VirtualLock, as that would also tell Windows that the entire file is needed in memory.

Better to use an explicit hint: PrefetchVirtualMemory on Windows and madvise on Unix.

I came up with this untested code snippet: #740 - needs more testing

@x02Sylvie

@x02Sylvie How much RAM do you have on your machine?

Hi,

My PR #801 will likely be merged soon. It includes a change that should theoretically fix the regression reported here, but I'd like to confirm that it actually does so.

If you're one of the people experiencing this issue, I'd love if you could test the PR and report how it affects performance. If possible, please do the following:

Before each test, either clear the Windows file cache as described here, or just reboot. Either way, the model is forced to be reloaded from disk.
Compare these three scenarios using the same model file:
1. Current master
2. PR branch, same command line as before
3. PR branch, but add --no-mmap to the command line
Optionally also compare to the codebase prior to the regression (though that would require a different model file).

Thanks in advance!

Update app.py (ggml-org#705)

This issue was closed because it has been inactive for 14 days since being marked as stale.

remove curious assertions (ggml-org#705)

Navigation Menu

Repository navigation

Windows page fault disk i/o slow on first load #705

15 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Windows page fault disk i/o slow on first load #705

Description

Activity

jart commented on Apr 2, 2023

anzz1 commented on Apr 2, 2023

fgdfgfthgr-fox commented on Apr 2, 2023

BadisG commented on Apr 2, 2023

CoderRC commented on Apr 2, 2023

PriNova commented on Apr 2, 2023

jart commented on Apr 2, 2023

CoderRC commented on Apr 2, 2023

CoderRC commented on Apr 2, 2023

slaren commented on Apr 2, 2023

jart commented on Apr 2, 2023

15 remaining items

CoderRC commented on Apr 2, 2023

patrakov commented on Apr 2, 2023

CoderRC commented on Apr 2, 2023

danielzgtg commented on Apr 2, 2023

CoderRC commented on Apr 2, 2023

CoderRC commented on Apr 3, 2023

comex commented on Apr 3, 2023

prusnak commented on Apr 3, 2023

prusnak commented on Apr 3, 2023

comex commented on Apr 7, 2023

github-actions commented on Apr 11, 2024

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions