4GB RAM limit in 32bit OS, PAE, DEP, 4GT

[®®®]

UPDATE 2008-02-13

Here's what Toshiba says:

4GB memory installed, but not all is available

With a total of 4GB (gigabytes) of memory installed in the computer, the amount of available memory (memory that is available to the operating system and to applications) will typically be less than 4GB. The amount of available memory will vary from model to model, depending upon the combination of several factors, including whether the CPU, system controller chip set, BIOS, and OS are of the 32 bit or the 64 bit variety.

If the computer has a 32 bit CPU, a chipset which is limited to 32 bit memory addressing, a 32 bit BIOS, or a 32 bit OS, then most of the 4th gigabyte of the computer's memory address space will necessarily be pre-allocated for PCI and graphics use, limiting the computer's total available memory to roughly 3GB.

If all of these components -- CPU, chipset, BIOS and OS -- fully support 64 bit memory addressing, then the computer's available memory can be greater -- nearly 4GB for some models. Computer manufacturers have limited control over the design of these components, which are produced by other manufacturers. As of August 2007, some of the chipsets used in Toshiba notebook computers -- including some AMD chipsets and the Intel 965 chipset -- support 64 bit memory addressing.

BIOS versions for the models listed above offer varying degrees of support for 64 bit memory addressing. For some models, relatively newer versions of the BIOS may increase somewhat the amount of available memory. Toshiba recommends periodically checking the Service and Support website at http://pcsupport.toshiba.com for newer versions of the BIOS for your computer.

Even without complete support for 64 bit memory addressing (by the CPU, BIOS, chipset and OS), the computer's performance may be significantly improved by expanding its memory to the maximum capacity of 4GB.

For more information, see the Microsoft Knowledge Base article on this topic: The system memory that is reported in the System Information dialog box in Windows Vista is less than you expect if 4 GB of RAM is installed

UPDATE 2008-01-29:

i turned on DEP and PAE over the bcdedit command but in task manager vista still shows me only 3.07 Gig Ram.

SP1 for VISTA will fix that cosmetic issue (as confirmed HERE):

With SP1, Windows Vista will report the amount of system memory installed rather than report the amount of system memory available to the OS. Therefore 32-bit systems equipped with 4GB of RAM will report all 4GB in many places throughout the OS, such as the System Control Panel. However, this behavior is dependent on having a compatible BIOS, so not all users may notice this change.

I'm reheating Pieter's findings over HERE regarding running a 64bit OS with 4GB or more of RAM and i add some technical links for the interested one to dig deeper into this quite important topic (yes, eventually we all have to switch to 64bit because of the Year 2038 issue :) ). I'm concentrating on XP and VISTA as these are the OS most of us use.

My approach is though a different one... mainly concentrating on 32bit OSs as these need some prerequisites enabled to use more than 4GB of memory.

Here's are the facts:

1. We distinguish between "Physical Memory Size Limits" (amount of RAM you add to your PC) and "Memory and Address Space Limits" (what _one_ process can actually use). This is important to understand why there is so much confusion amongst people.

2. Physical RAM Limits for each OS are officially stated over @ this recently updated MSDN site.

• Physical RAM Limits Windows XP 32 & 64 bit
  
• Physical RAM Limits VISTA 32 & 64 bit (all editions)
  

Adress Space Limits are stated over HERE.

The first mythbuster:

Limits on physical memory for 32-bit platforms also depend on the Physical Address Extension (PAE), which allows 32-bit Windows systems to use more than 4 GB of physical memory.

3. XP and VISTA 32bit OS have a limit (without using DEP) of 4GB (exception: Starter Editions, which are crippled anyways). 64bit editions of XP and VISTA overcome this limit without any additional tools or settings. Support for more than 4GB of memory is enabled by default.

Summarized: running a 32-bit edition of Windows XP or VISTA supports only 32-bit memory addressing and therefore is able to access only 4 GB of physical memory.

4. And again:

The Physical Address Extension (PAE) allows 32-bit Windows systems to use more than 4 GB of physical memory. PAE also enables several advanced system and processor features, such as hardware-enabled Data Execution Prevention (DEP), Non-Uniform Memory Architecture (NUMA) and the ability to add memory to a system while it is running (hot-add memory), so it can also be used on computers that have less than 4 GB of memory. PAE is supported only on 32-bit versions of Windows; 64-bit versions of Windows do not support PAE.

On most computers, PAE is disabled by default. (PAE is enabled by default only if DEP is enabled on a computer that supports hardware-enabled DEP, or if the computer is configured for hot-add memory devices in memory ranges beyond 4 GB.)

PAE increases the physical address size from 32 bits to 36 bits. This increases maximum physical memory size from 4 GB to 64 GB. The 32-bit size of virtual address is not changed, so regular application software continues to use instructions with 32-bit addresses.

With PAE enabled, the operating system moves from two-level linear address translation to three-level address translation. The extra layer of translation provides access to physical memory beyond 4 GB.

To enable PAE, use the /PAE switch in the Boot.ini file. To disable PAE, use the /NOPAE switch.

More technical information regarding PAE can be found over HERE.

5. While the /PAE switch allows the operating system to access and use more than 4 GB of physical memory, there's another switch called "4-gigabyte tuning" (4GT) which extends the 32-bit user virtual address space from 2 GB to up to 3 GB. 4GT can be used with or without PAE.

On 32-bit editions of Windows, applications have 4 gigabyte (GB) of virtual address space available. The virtual address space is divided so that 2 GB is available to the application and the other 2 GB is available only to the system.

The 4-gigabyte tuning (4GT) feature, formerly called 4GT RAM Tuning, increases the virtual address space that is available to the application up to 3 GB, and reduces the amount available to the system to between 1 and 2 GB.

To enable 4GT in 32bit XP, add the /3GB switch to the Boot.ini file.

To enable 4GT in 32bit VISTA, use BCDEdit /set command to set the "increaseuserva" boot entry option to a value between 2048 (2 GB) and 3072 (3 GB).

6. DEP:

There are two types of DEP:

• Hardware-enforced DEP enables DEP for both kernel-mode and user-mode processes. It must be supported by the processor (most recent CPU have this feature, check the specs over at INTEL or AMD if unsure) and the operating system.
  
• Software-enforced DEP enables DEP only on user-mode processes. It must be supported by the operating system.
  

When DEP is enabled on a computer with a processor that supports hardware-enforced DEP, Windows automatically enables PAE (as hardware-enforced DEP requires PAE). It then ignores any PAE-disabling boot parameter which may have been set by the user.

Further information regarding how to enable DEP and PAE in XP (boot.ini) and VISTA (BCDedit) can be found over HERE.

The most important part for using more than 4GB is apparantly to enable the DEP feature in your computer's BIOS (look for a setting called XD or NX under the "Security" or the "CPU information page" or whatever it is called in your setup). Some manufacturers disabled this for some reason. Also check if your mainboard (chipset) can take more than 4GB RAM :)

[mobilenvidia]

Very good write up Rene :)

Clears up a few things.

What is limiting us really is the Motherboard chipset limitations.

My i9400 BIOS only sees 3.25GB and hence Vista only sees this as well.

My Vist ax64 Ultimate is capable of addressing way more RAM.

The i9400 has the i945 chipset and I do believe the i1720 has the i965 chipset which can see more than 3.25GB.

So all those CPU features of addressing more than 4GB is of little use if your Chipset doesn't allow for it.

Notebook BIOS's don't have much option for setting change, so we are stuck with what were given.

[mobilenvidia]

UPDATE

Just had a quick look in the BIOS.

It sees 4GB but can only use 3.25, with an explaination that the difference is used by system resources.

Now I assume the chipset cannot map these resources above the 4GB memory limit.

Newer chipsets must be able to put the resources somewhere way higher in the address range.

[Fabrice Roux]

For those interested in Hardware DEP and its limitations, I have a blog post about HDEP.

[Tomcheck]

i turned on DEP and PAE over the bcdedit command but in task manager vista still shows me only 3.07 Gig Ram.

[mobilenvidia]

Nice blog site.

1/2 dome is distracting though as it brings back fond memores of a rather awsome feat of nature :)

[®®®]

i turned on DEP and PAE over the bcdedit command but in task manager vista still shows me only 3.07 Gig Ram.

SP1 for VISTA will fix that cosmetic issue (as confirmed HERE):

With SP1, Windows Vista will report the amount of system memory installed rather than report the amount of system memory available to the OS. Therefore 32-bit systems equipped with 4GB of RAM will report all 4GB in many places throughout the OS, such as the System Control Panel. However, this behavior is dependent on having a compatible BIOS, so not all users may notice this change.

[hola]

"For example, consider a computer that has an Intel 975X chipset that supports 8 GB of address space. If you install 8 GB of RAM, the system memory that is available to the operating system will be reduced by the PCI configuration requirements. In this scenario, PCI configuration requirements reduce the memory that is available to the operating system by an amount that is between approximately 200 MB and approximately 1 GB. The reduction depends on the configuration."

http://support.microsoft.com/kb/929605

Edited January 29, 2008 by hola

[envydyauser]

Very good summary.

However, for the record, let'sput things in context.

Summarized: running a 32-bit edition of Windows XP or VISTA supports only 32-bit memory addressing and therefore is able to access only 4 GB of physical memory.

This "only" mentality is a direct result of Microsoft marketing. 64-bit at home is still, in 2008, a purely academic discussion.

64-bit wasn't born yesterday : server-side, it's been around about 10 years, which is normal given the memory requirements for business server apps.

However, I defy any home user to get anywhere near 4 GB of RAM usage.

OK, given the memory footprint growth of Windows versions over time (approx 64MBfor WinNT4, 128MB for Win2K, 256MB for WinXP, I don't know about Vista) I can concede that a home system now needs 2 GB to be on the safe side (I have added my 2nd GB just for that reason).

However, when I run Photoshop (with a dozen large pics) Adobe Premiere with all its filters, Virtualdub, Firefox w/a dozen tabs, etcetera, I still use less than 1 GB RAM. (XP)

cheers

envydyauser

[PG.]

SP1 for VISTA will fix that cosmetic issue (as confirmed HERE):

i have SP1 refresh 2 . Vista says in the welcome screen i have 4GB, but taskamanager says 3GB (3070) everest , procesexpleore also and it looks like it cannot use more.

so i tried this , but it didnt change anythig.

PAE: i turned PAE via BCDEdit /set PAE forceenable

DEP: (in options performance etc is DEP enabled for system programs and services)

(4GT then i tried bcdedit /set IncreaseUserVA 3072)

BIOS says i have only 3GB memory, but ubuntu uses 4GB (3.85GB something) , also memtest can test the whole 4GB.

Does anybody know what am i doing wrong?

[Tomcheck]

for me the same. Bios and Vista SP1 RC say 4GB, Taskmanager sais 3.07GB.

[whitetigerx7]

I can't use 4GB anyway with my Laptop as my BIOS limits it to 2GB but anyway 4GB for anything is overkill.

[mobilenvidia]

You'd be surprised how much Vista eats memory especially 64bit which loads both 32 and 64bit cores.

You try editing a full DVD movie (even a single layer) with 2GB or less, 7zip for me uses 2.3GB just to compress the drivers to what you see hosted here.

The big issues for getting past 4GB is the system resources being able to be mapped past the 4GB barrier.

Once this is done then all the 4GB is available to the OS and anything past 4GB will also be available.

Obviously newer chipsets can do this as can XEON chipsets.

I would think Intel keeps costs down on mainstream MB's or marketing ploy as you have to upgrade to a new MB to get 4GB+ (well 3GB+ really)

So if your BIOS sees 4GB but only allocates 3GB usable then I doubt any OS tweaks will allow the full 4GB to be available.

As the system resources will have nowhere to go, similar to the 640KB DOS limit of old with 1MB machines.

[®®®]

The P35 chipset can do so (though it#s a desktop chipset). I have 6GB installed for testing reasons and Vista SP1 just happily recognizes everything without a glitch. I guess the X38, X48 and generally all current x3x INTEL chipset have no problem. It#s apparantly the older generation of 9xx chipsets.

[PG.]

...So if your BIOS sees 4GB but only allocates 3GB usable then I doubt any OS tweaks will allow the full 4GB to be available.

As the system resources will have nowhere to go, similar to the 640KB DOS limit of old with 1MB machines.

images say more as words right :)

memtest:

linux (previous posted too):

• The OS gives a sh.it about BIOS :)
  BIOS shows only 3GB (see second picture) and i doubt he allocates 4GB or?
  Windows first picture knows i have 4GB and Linux x64 actually uses them(the whole 4GB, last picture)
  Memtest test them (the whole 4GB) and they PASS
  

Taskmanager

Dont know why but sometimes the availaible memory in PE or taskmanager or everest is different and differs like you can see by 0.5gb. Does someone know what the cause could be?

As for the overkill .

FX has on start 500MB memory taken (on mine PC many tabs , for work not just "fun") that would be 1/4 of your (whitetigerx7) 2GB and thats only one aplication .

Edited January 30, 2008 by PG.

[®®®]

Here's what Toshiba says:

4GB memory installed, but not all is available

With a total of 4GB (gigabytes) of memory installed in the computer, the amount of available memory (memory that is available to the operating system and to applications) will typically be less than 4GB. The amount of available memory will vary from model to model, depending upon the combination of several factors, including whether the CPU, system controller chip set, BIOS, and OS are of the 32 bit or the 64 bit variety.

If the computer has a 32 bit CPU, a chipset which is limited to 32 bit memory addressing, a 32 bit BIOS, or a 32 bit OS, then most of the 4th gigabyte of the computer's memory address space will necessarily be pre-allocated for PCI and graphics use, limiting the computer's total available memory to roughly 3GB.

If all of these components -- CPU, chipset, BIOS and OS -- fully support 64 bit memory addressing, then the computer's available memory can be greater -- nearly 4GB for some models. Computer manufacturers have limited control over the design of these components, which are produced by other manufacturers. As of August 2007, some of the chipsets used in Toshiba notebook computers -- including some AMD chipsets and the Intel 965 chipset -- support 64 bit memory addressing.

BIOS versions for the models listed above offer varying degrees of support for 64 bit memory addressing. For some models, relatively newer versions of the BIOS may increase somewhat the amount of available memory. Toshiba recommends periodically checking the Service and Support website at http://pcsupport.toshiba.com for newer versions of the BIOS for your computer.

Even without complete support for 64 bit memory addressing (by the CPU, BIOS, chipset and OS), the computer's performance may be significantly improved by expanding its memory to the maximum capacity of 4GB.

For more information, see the Microsoft Knowledge Base article on this topic: The system memory that is reported in the System Information dialog box in Windows Vista is less than you expect if 4 GB of RAM is installed

[mobilenvidia]

Vista SP! is deceiving.

Beta says 3.25GB available and reports this also in the Task manager.

SP1 RC reports 4GB available but Task manager reports 3.25GB

In the end the BIOS decides what memory is availabe with the components available.

My i9400's i945 Chipset = 32bit address extensions = 4GB limit (includeding mapping of system resources)

i965 must have full EMT64 addressing and be able to address more than 4GB, the system resources can be mapped past what ever RAM is installed.

[PG.]

OK. If you still say its on the BIOS (mine says I have only 3GB). How come that 64 BIT OS(as i posted earlier) can use the whole 4GB (memtest also)?

Btw, I wouldn't be sure that you can run i945 with 4GB, doesn't probably even start. What I have browsed forums, the max on this chipset is 2.5GB (2GB stick + 512MB).(FSC for sure)

Edited February 14, 2008 by PG.

[®®®]

Here's a well-done summary which goes into detail even more:

http://www.dansdata.com/askdan00015.htm

It seems that two 512MB SLI-driven cards eat up 1GB of RAM :)

[Darmdorf]

Come on guys, it's actually pretty simple - assuming you got in on the ground floor of TTL logic and microprocessors in the mid-70's. :P The modern chipset in your mother board determines the actual addressing bandwidth to the components, be they system memory or memory-mapped devices like I/O chips, graphic cards, or what ever. 32-bit = 4GB. Just 'what' can be addressed within that range depends upon the systems architecture. Traditionally the CPU determined the address range, but today the CPU is flanged onto the board's chipset to achieve asynchronous [i.e. overlapping] access to the various components. The chipset is not one chip but rather usually 2 disguised as the memory manager or famous Northbridge in 'Intel-speak' which handles memory access to system memory, and the I/O manager or Southbridge which handles the periphery. AMD includes the memory management functions directly on-chip and thus often gains a performance leg up on Intel.

Way back in the early days of 8-bit processors engineers strived to maintain the full available addressing range for memory access (after all, 256 bytes isn't exactly the world)! To that end I/O devices were placed into their own private memory space, and accessed through special instructions [port I/O] one byte at a time, giving birth to port-mapping. The address value placed on the address bus for I/O was the same as when performing normal memory accesses, but the port I/O instructions caused a special I/O line to also be triggered which was used to close access to the memory chips, but open access to any addressable I/O devices. That meant you could have just as many I/O ports as the CPU's address range would support. Some CPU's even went so far as to directly implement banked-memory. That was the magic of writing the bank number to a special bank I/O port, and then addressing that 'level' of memory selected through the bank value with the full addressing range. We went wild and could address 64KB with an 8-bit processor. :) As processors developed further this bank selection was dubbed 'segmentation', and was still a nasty hurdle. Eventually forms of virtual addressing were implemented, which opened the way to the OS ring architecture and various protection schemes.

The problem of re-using the processor's address range for both memory and I/O chores is still with us today. In fact the problems involved have become worse, since within modern operating systems the use of 'true' I/O port accesses are the singular privilege of the operating system drivers. Ports are of course 'dis-functional' for the needs of modern programming, since only a single byte or word can be accessed by a port instruction. That is just a bit too thin to be moving megabytes of data to hard drives or graphic memory! Historically speaking the re-use of the processors address range for both memory and I/O purposes died with the arrival of 16-bit hardware. Surprised? The engineers were almost certain that CPU's would NEVER be given so much memory as to make full use of the address range. The I/O chores also changed drastically, no longer restricted to pushing a byte here, and a byte there. Applications now needed to quickly move entire blocks of data to new block-oriented output devices - screens, tapes, and even hard drives. Since the address 'real estate' was almost as endless as the horizon, the solution to those new I/O chores was to implement the previously mentioned 'memory-mapping'. This approach carved up segments of the address space into 'plots' directly at the hardware level where the private memory areas of input or output devices could be blended into the CPU's view. Now it was a snap to copy entire arrays directly to or from I/O devices with simple string instructions [or DMA - direct memory access transfers]. For a program there was no difference between writing to a normal data variable or array, and the memory of a graphic adapter. This engineering solution is simple, elegant, and easy to implement -- but at the cost of addressable space for system memory.

So....now down to the theme of this thread.

That priceless 4GB of memory that you've installed is of course still fully addressable by the CPU. :) During the boot process the BIOS has merely instructed the chipset [Northbridge] to blend in the private memory areas of those lovely I/O devices you've installed and enabled, right OVER your ram, in a standardized mapping which applies to your particular type of machine [PCs are different than Apples]. The more private memory that expensive graphic card holds, the less system memory is left 'in view' and available to those little operating system processes.

The installed memory is ALWAYS available to special memory test programs such as memtest, because they possess the ability to manipulate the chipset just like the BIOS, but this time to blend out any memory-mapped devices, leaving the memory in full view for the CPU. That is why the BIOS and some truthful OS's report the full memory complement, while the task manager only reports what is left after the BIOS's memory-mapping strategy and thus available for process mapping. This is of course a slight simplification regarding some details, but it serves to explain what you observe. The analogy would be --- do you consider the bathroom as living space?

So...now that you understand what is going on inside your favorite piece of hardware, you may be tempted to opt for just 3GB of system memory? DON'T DO IT! A 1 and a 2 GB stick will have a significant performance hit, due to the lousy interleave arrangement. Interleave is the Northbridge's ability to 'double-dip', i.e. alternately access 2 banks of memory and thus allow the memory chips to 'pause' before another access -- blending out the recharge phase. As you can surmise, when the 2 banks are of unequal size, the big guy gets hit twice as often --- again & again & again ---- OUCH!

Oh yes, PAE might be able to virtually map the memory-mapped devices to an area 'above' the 4GB boundary to that no system memory is over-shadowed.

Oh yes, just one more thing....Now that you have a firm understanding of the subject, please go back and reread this thread. I bet all that gibberish at the outset will now be a piece of cake. Insight makes one warm & cozy.

[Guest Guest]

I think what you guys are missing is that physical address extension (PAE) is needed to even impliment DEP; it has to recognize those memory addresses above 4GB to BLOCK them. DEP is meant to block anything pointing at a physical address higher than the x32 limit of 4GB, like in the case of an attack hidden in a buffer overflow. The trick here, and I'm surprised that flippin Steve Wozniak here doesn't get it when I just learned about it tonight and I'm pretty much a computer dunce, is to DISABLE DEP while still enabling his lovely little partner PAE. That way thecomputer recognizes addresses above 4GB and they're NOT blocked! They're free for your memory to spill over into, but it can cause some compatability issues.

[coolcop06]

well i followed all the steps and I THINK i have done everything correctly, but it still shows 3581 MB total ram available...

can someone help me? is there something im missing?

does anyone have actual proof (like before and after screenshots) that this actually works??

thanks

[Michael Marley]

i965 must have full EMT64 addressing and be able to address more than 4GB, the system resources can be mapped past what ever RAM is installed.

I can confirm that the 965 supports 4gb of RAM. My system has this much, and all of it appears in Windows XP x64. (I have not tried Vista, since I got the RAM.) Still, I remember reading that the 965 has 40-bit memory addressing, not the full 64-bit, but still enough for 1200 GB of RAM.

[tcsenter]

well i followed all the steps and I THINK i have done everything correctly, but it still shows 3581 MB total ram available...

can someone help me? is there something im missing? does anyone have actual proof (like before and after screenshots) that this actually works??

Unfortunately, much of the information provided here is wrong, or at least a good four years out-dated, due to Microsoft's documentation on PAE being out-dated.

As of Service Pack 2 for Windows XP (2004), Microsoft modified (a.k.a. crippled) the PAE kernel by limiting it to a maximum of 4GB physical RAM. What that means is, PAE on Windows XP gives you the exact same RAM limitation as without the PAE kernel. Microsoft states it was forced to do this due to the significant number of 32-bit device drivers it encountered during testing that are not written to properly cope with addresses greater than 32-bit, causing bad things to happen under PAE mode such as driver and OS crashes.

Microsoft decided to apply this modified PAE behavior to all 32-bit Vista editions as well. This is a hard-coded limit in the kernel MMU. There is no combination of boot switches or any other tricks to make 32-bit XP or Vista support more physical RAM with PAE than it does without PAE. The only thing that would accomplish this is to rewrite a significant portion of the OS kernel.

Only select Microsoft Server SKUs support more than 4GB physical RAM via 36-bit PAE, such as Server 2003 and Server 2008 Enterprise Editions. Entry-level SKUs such as Server 2003 and 2008 "Standard" Editions have the same 4GB limitation as XP and Vista, with or without PAE.

In order to use more than 3.2GB ~ 3.5GB RAM on any Windows desktop OS, you'll need a 64-bit version and 64-bit ready hardware. 32-bit Linux distros can do it as well using PAE mode, subject to hardware limits.

[coolcop06]

thank you so much for clarifying this issue. i was pretty sure i had followed all the steps now and that it wouldnt work no matter what i did, but its nice getting the reason why its not working.

thanks again!