Discussion
I'm on Macintosh, or Win32.
I say:
void *x = malloc(1024*1024*4);
Or perhaps I say:
void *x = malloc(1024*1024*1024);
That's ordinary malloc. Just pointed clang or MSVC at a .c file and said go.
Do you expect that these calls allocate from heap, or do you expect they allocate from virtual memory?
Do you expect the answer to differ between Macintosh and Windows? Differ between Windows 7 and Windows 11? Differ between the first and second code snippet?
@mcc@mastodon.social
To my knowledge, this is a misunderstanding. Virtual memory means the memory addresses you look up are altered on the fly to keep different processes from seeing each other's memory, which on any OS made in the last 25 years happens to all memory, including the heap.
@hatzka there is malloc(), and there is mmap(MAP_ANONYMOUS). I am trying to figure out if there is literally any difference between the two in the year 2026. Even if it's an insignificant difference. I am trying to figure out the difference. My *assumption* is that if nothing else using the latter may sometimes avoid memory fragmentatoin.
@mcc the idea of distinguishing between heap and virtual memory doesn’t make sense to me? It’s always going to be both of those things. All the heap means to me is the dynamic lifetime memory outside the stack, and all memory that the OS manages is going to be virtual memory.
@porglezomp It doesn't matter if the two are interchangeable; I simply want to know.
@mcc what would be the difference between heap and virtual memory in that case? Managing memory or just memmapping directly?
@Ronflaix My understanding is processes have (1) a linear, growing (sbrk) memory space that the heap gradually grows into and (2) other blocks requested directly from the virtual memory page allocator. I may be incorrectly assuming Windows works like POSIX here.
Actually, wait, here's the question I should have asked.
- I have just allocated a large buffer with malloc. For an unimportant reason, I am curious whether it was allocated off the heap or via the OS-local equivalent of mmap(MAP_ANONYMOUS). Is there a way, in code, to test which one I got? Does the answer differ between Linux/MacOS/Windows?
@mcc 1) based on my work with audio + drivers, I always assume allocated memory is virtual and unmapped until used, even though this is frequently not the case, and will move/swap at any time. The answer varies based on OS, runtime lib, and buffer size (and version of libraries and OS and amount of free memory etc depending on OS and hardware platform).
Back issues of PoC||GTFO highly recommended as reading-for-pleasure in this area. :)
@mcc reading through various replies to this I think I now understand what you are trying to ask, so I will answer that instead.
Regardless of the method used to get there, all pointers in userspace are virtual addresses that map to some range of physical addresses. Outside of niche OS configurations, no OS makes any guarantees about avoiding memory fragmentation across page boundaries (but anything within a page will be contiguous because of how processor page tables work). (1/2)
@mcc malloc_claimed_address() on modern Darwin
@Catfish_Man Imagine me doing the "sickos yes" thing to the computer screen at learning about this function
@mcc@mastodon.social "the heap" is just locks of memory allocated via mmap(MAP_ANONYMOUS)
@erincandescent I was once told a story about POSIX functions named brk and sbrk, and the conventional heap slowly growing into the brk space. Is this outdated?
Or do you mean even brk/sbrk are fundamentally just asking the virtual memory page allocator for pages, when you get inside of the libc?
@mcc@mastodon.social traditionally sbrk/brk space grows up from the end of your app's bss into the stack. when dynamic libraries came around, that space wasn't generally one big hole any more. this dates back from an era where virtual memory & paging didn't exist.
sbrk/brk still exist but modern allocators typically just use mmap as the more general api. if they are using sbrk (for historical reasons, mainly), then they treat sbrk and mmap as interchangable.
sbrk is a bad api because it doesn't provide a way to give holes back to the kernel
@mcc @erincandescent the wikipedia page for brk/sbrk notes that mac os x emulates sbrk and its implementation of brk always returns an error.
@mcc@mastodon.social i guess the real difference is "was anything else placed in that mmap block?"
@mcc there's no distinction between the heal and virtual memory really, all memory is allocated through the virtual memory APIs. Whether or not any particular allocation hits the syscall to assign more virtual memory depends on the allocator, its configuration, and the state of the allocator (globally and on the allocating thread)
@mcc this includes stuff like whether your process on windows opts in to the "new" allocator
@dotstdy This is very interesting to me, I have not heard of the "new" allocator, how would one tell which one is using? Or I guess what I really mean is, what keyword should I search for on learn.microsoft.com to find out which CMake/Visual Studio flags would govern tihs?
@mcc Just to counter some of the bad answers you're getting: heap and VM allocation *are* very different, because an allocation large enough to cross the threshold to directly using mmap (or platform equivalent) may have a virtual address chosen to use TLBs more efficiently (e.g large pages). It might not even participate in the usual heap structs, and instead the address range or top bits to flag that it's a mapping. It will also very likely return a large-page-aligned address.