How to limit the address space of 32bit application on 64bit Linux to 3GB?

Is it possible to make 64bit Linux loader to limit the address space of the loaded 32bit program to some upper limit?

Or to set some holes in the address space that to not be allocated by the kernel?

I mean for specific executable, not globally for all processes, neither through kernel configuration. Some code or ELF executable flags are examples of appropriate solution.

The limit should be forced for all loaded shared libraries as well.

Clarification:

The problem I want to fix is that my code uses the numbers above 0xc0000000 as a handle values and I want to clearly distinct between handle values and memory addresses, even when the memory addresses are allocated and returned by some third party library function.

As long as the address space in 64bit Linux is very close to 4G limit, there is no enough addressing space left for the handle values.

On the other hand 3GB or even less is far enough for all my needs.

Answer

OK, I found the answer of this question elsewhere.

The solution is to change the “personality” of your program to PER_LINUX32_3GB, using the Linux system call sys_personality.

But there is a problem. After switching to PER_LINUX32_3GB Linux kernel will not allocate space in the upper 1GB, but the already allocated space, for example the application stack, remains there.

The solution is to “restart” your program through sys_execve system call.

Here is the code where I packed everything in one:

proc ___SwitchLinuxTo3GB
begin
        cmp     esp, $c0000000
        jb      .finish                 ; the system is native 32bit

; check the current personality.

        mov     eax, sys_personality
        mov     ebx, -1
        int     $80

; and exit if it is what intended

        test    eax, ADDR_LIMIT_3GB
        jnz     .finish                         ; everything is OK.

; set the needed personality

        mov     eax, sys_personality
        mov     ebx, PER_LINUX32_3GB
        int     $80

; and restart the process

        mov     eax, [esp+4]          ; argument count
        mov     ebx, [esp+8]          ; the filename of the executable.
        lea     ecx, [esp+8]          ; the arguments list.
        lea     edx, [ecx+4*eax+4]    ; the environment list.

        mov     eax, sys_execve
        int     $80

        ; if something gone wrong, it comes here and stops!
        int3

.finish:
        return
endp