mirrors/mimalloc
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
1,067 Commits 21 Branches 59 Tags 7.4 MiB
1b571aea06
Commit Graph

686 Commits

Author SHA1 Message Date
daan
8422ab125d improve messages; fix reset size calculation on large pages 2020-01-31 17:28:26 -08:00
daan
724602b78b enable page-reset by default 2020-01-31 17:27:45 -08:00
daan
4531367de2 fix padding check for aligned allocation; improve perf for small aligned allocations 2020-01-31 13:20:02 -08:00
daan
f820fae858 merge from dev; add retired collect on abandon 2020-01-30 07:24:35 -08:00
daan
a7c69ccbea fix stat accounting of segments with huge blocks 2020-01-30 06:25:42 -08:00
daan
ed1c8a203a improve performance with tls recursion counter 2020-01-29 23:08:52 -08:00
daan
03b363a1c2 first working tls on macOS using interpose; still slow 2020-01-29 22:46:44 -08:00
daan
7ff3ec2bf7 use EFAULT for buffer overflow and call abort in debug mode (as well as secure mode) 2020-01-29 17:25:40 -08:00
daan
5d212d688f add MI_PADDING build option to add padding to each block to detect heap block overflows 2020-01-29 17:10:57 -08:00
daan
b3dae128de Merge branch 'dev' into dev-exp 2020-01-27 23:15:37 -08:00
daan
9c166d88f0 increase retire page size 2020-01-27 23:15:24 -08:00
daan
42586de104 fix is_zero setting in regions 2020-01-27 23:13:57 -08:00
daan
e628fc7067 cleanup reclaim logic 2020-01-26 12:39:11 -08:00
daan
d4927adddc add extra assertion that all segments are free on thread termination 2020-01-25 14:30:02 -08:00
daan
394b796ea0 fix over-eager page reset in segment reclamation 2020-01-25 13:43:56 -08:00
daan
4faf412f53 move 'memory.c' to 'region.c' 2020-01-25 13:28:49 -08:00
daan
7785139201 fix warning on gcc on attribute ignore in templates 2020-01-25 13:28:24 -08:00
daan
62b8fb26b1 fix freeing of segments on forced reclaim 2020-01-25 13:27:47 -08:00
daan
cb433d272a Merge branch 'dev' into dev-exp-abandon 2020-01-25 13:03:06 -08:00
daan
f4630d43a7 allow reset on large pages; check commit status before reset 2020-01-25 12:49:14 -08:00
daan
5e32d00aab add visit count to abandoned to limit list length 2020-01-25 12:26:08 -08:00
daan
3ec691cfb4 Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev 2020-01-25 11:20:17 -08:00
David Carlier
2b667bd3ae enable arc4random abi under apple 2020-01-25 14:47:09 +00:00
daan
072ceb2efe Merge branch 'dev' into dev-exp-abandon 2020-01-24 20:20:55 -08:00
daan
ecece57284 fix bug in committed check in arena allocation 2020-01-24 20:20:43 -08:00
daan
47300eeda3 avoid memset 2020-01-24 20:17:33 -08:00
daan
b31bc52618 add cache alignment directives for contended variables 2020-01-24 19:02:13 -08:00
daan
e070eba112 fix tagged null encoding, search segment cache before reclaim 2020-01-24 16:31:18 -08:00
daan
9a400ecce9 Merge branch 'dev' into dev-exp-abandon 2020-01-24 15:48:34 -08:00
daan
4ae51096ec add warning on region exhaustion 2020-01-24 15:45:03 -08:00
daan
28c14d99c3 clean up comments 2020-01-24 11:03:12 -08:00
daan
e68293741e fix assertion, add check for page committed before doing reset 2020-01-23 21:44:32 -08:00
daan
58fdcbb0cd fix bug in collect where has_page was not set on free pages 2020-01-23 21:37:14 -08:00
daan
4a2a0c2d50 initial abandon based on fine-grained reclamation 2020-01-23 19:50:35 -08:00
daan
12701b1aac do not reclaim segments on collect 2020-01-23 15:48:51 -08:00
daan
76e727f7d1 fix assertion on page destroy 2020-01-23 15:39:38 -08:00
daan
0316859e06 improve codegen for mi_free 2020-01-23 10:03:03 -08:00
daan
66818bf632 use atomic yield on delayed-freeing; clarify code 2020-01-23 09:57:55 -08:00
daan
3bbbe6c686 enable atomic yield when delayed_freeing is encountered 2020-01-22 20:12:42 -08:00
daan
0193a15f7e nicer message on huge OS page reservation 2020-01-22 19:46:43 -08:00
daan
e3b16fe4ef reduce type casts in atomic operations 2020-01-22 19:44:54 -08:00
daan
3957b2fd28 add mi_new_realloc(n) to support C++ style reallocation that raises std::bad_alloc on out-of-memory 2020-01-20 15:41:56 -08:00
daan
b77be05e40 only collect retired at fresh page allocation 2020-01-20 12:14:34 -08:00
daan
e8d7c80c74 fix build warnings on linux 2020-01-19 17:33:36 -08:00
daan
41e717c2e0 fix assertion in mi_block_zero_init (issue #194) 2020-01-18 20:30:12 -08:00
daan
dc58388968 Add ability to register custom error function called on various error conditions; including ENOMEM 2020-01-17 19:59:55 -08:00
daan
6dd636d82d improve STL allocator using mi_new_n and removing unused parameter names; follow up from pr #193 and #188 2020-01-17 15:41:52 -08:00
daan
6b7356a10a make mimalloc compile with the highest warning level on msvc 2020-01-16 18:12:01 -08:00
daan
c6037ac8f2 fix assertion index 2020-01-16 16:56:02 -08:00
daan
3f8b8b7a55 initialize thread_delayed_free field atomically 2020-01-16 16:45:34 -08:00
daan
da709bcf11 test for gcc version for aligned new/delete overrides 2020-01-16 16:30:44 -08:00
daan
7a98a461a3 fix type of bitmap field to avoid C++ error on older gcc 2020-01-16 16:21:59 -08:00
daan
7a9502973d rearrange STL allocator code: remove pragma, ifdef for C++11 2020-01-16 15:57:11 -08:00
daan
4e91eab8fc specialize mi_mallocn for count=1 2020-01-16 14:12:37 -08:00
daan
8d8f355ed0 add option to reset eagerly when a segment is abandoned 2020-01-16 11:25:02 -08:00
daan
b8072aaacb fix debug build 2020-01-16 03:54:51 -08:00
daan
9629d73188 fix options 2020-01-15 18:07:29 -08:00
daan
7a1e86fc20 merge from dev 2020-01-15 18:03:37 -08:00
daan
ad32eb1dfb eager collect on page reclamation 2020-01-15 17:57:19 -08:00
daan
0099707af9 use delayed free for all pages; reduce size of the page structure for improved address calculation 2020-01-15 17:19:01 -08:00
daan
c9b5ac80b3 update page reset queue to just do delayed page resets 2020-01-15 12:00:44 -08:00
daan
783e3377f7 add output argument to stat printing 2020-01-15 10:53:54 -08:00
daan
0956a05bf6 add argument pointer to the register deferred free callback 2020-01-15 10:21:32 -08:00
daan
f92a2a7264 add argument pointer to the register output routine 2020-01-15 10:18:32 -08:00
daan
941c55ee42 wip: first implementation of page free list on segments for effecient delayed page reset 2020-01-14 21:47:18 -08:00
daan
65f4f5144b fix out-of-bounds error in huge OS page bitmap 2020-01-13 17:06:25 -08:00
daan
5f61a9e896 add mprotect error when the mmap limit might be reached in secure mode (see issue #77) 2020-01-09 17:52:28 -08:00
daan
8f75444e7a fix windows debug build at MI_DEBUG=2 2020-01-08 23:21:32 -08:00
daan
12ef2816ed fix bug exposed by commit 59fa286 where reclaimed pages could be stuck to NEVER_DELAYED 2020-01-08 19:00:03 -08:00
daan
683d8998d4 fix potential A-B-A problem with segment abandonment; noticed by Manual Poeter and Sam Gross 2020-01-08 17:45:38 -08:00
daan
50b3f6d7ae fix assertion 2020-01-08 16:33:45 -08:00
daan
d4ab0ff08c fix timeout on huge page reservation if set to 0 2020-01-07 14:15:37 -08:00
daan
4223caac0f on Linux dynamically detect if getrandom is supported and fall back to /dev/urandom if needed 2020-01-06 22:08:21 -08:00
daan
d8d69c2c94 disable MAP_NORESERVE on huge pages 2020-01-05 22:07:16 -08:00
daan
45582d1fb5 revert a2a9230 (remove empty page removal on search): this is not generally valid when concurrent frees do not always add to thread_delayed_free. 2020-01-05 13:58:49 -08:00
daan
59fa286294 fix bug where continue would wrongly exit the do-while loop for delayed freeing 2020-01-05 13:52:07 -08:00
daan
a2a9230ad6 remove empty page removal on page search (no longer needed with retired collection and delayed freeing) 2020-01-03 22:52:52 -08:00
daan
d596f08569 fix thread_free read in assertion 2020-01-03 22:06:27 -08:00
daan
2b108c8748 increase retire expiration to 4 2020-01-03 21:39:18 -08:00
daan
eeb623e6af increase retire limit, collect retired pages 2020-01-03 17:06:41 -08:00
daan
9629a0190f fix eager commit on large pages (issue #182) 2020-01-02 17:25:00 -08:00
daan
1b5a08cd25 remove unused parameter in check double free 2020-01-02 17:24:32 -08:00
daan
384808e340 Merge branch 'dev-random' into dev 2019-12-28 16:19:45 -08:00
daan
fc3e537bd4 improve double free detection with faster same page check 2019-12-28 15:28:13 -08:00
daan
77134e1ad0 update free list encoding to stronger formula with addition last 2019-12-28 15:17:49 -08:00
daan
e3391d9a53 stronger encoding of free lists using two keys per page 2019-12-28 00:57:42 -08:00
daan
ce02986d56 variable renaming 2019-12-27 22:30:23 -08:00
daan
6f0b35d513 merge from master 2019-12-24 10:44:49 -08:00
daan
e05a1edc03 fix large OS page size on Linux (issue #184, due to fix for #179) 2019-12-24 10:32:44 -08:00
daan
ba87a39d9f updated random cookie generation using OS primitives and chacha20 2019-12-22 19:37:49 -08:00
daan
f9b942d80d fix compilation of region descriptor on 32-bit 2019-12-22 17:08:46 -08:00
Daan Leijen
e31e609414 add preload check in option initialization (issues #179) 2019-12-01 01:03:39 -08:00
daan
36d168a2d9 add preload check to options initialization 2019-12-01 00:03:35 -08:00
daan
a407f35c64 add arena.c into the static override object 2019-12-01 00:01:14 -08:00
daan
1d998af854 clean up options; make secure work with eager_page_commit 2019-11-25 10:47:17 -08:00
daan
1643273357 fix unix bug in decommit size 2019-11-25 10:11:29 -08:00
daan
c6df7a199c experimental eager page commit option 2019-11-24 22:02:40 -08:00
daan
4452431b6c reenable segment cache and fix initial segment commit 2019-11-24 15:25:19 -08:00
daan
727d33b96f more precise memory reset 2019-11-24 14:40:47 -08:00
daan
4a0d35afd0 improve secure guard page allocation to work with non-eager commit 2019-11-23 11:59:19 -08:00
daan
6568059cc6 merge from dev 2019-11-21 16:28:28 -08:00
daan
74dbfc30be improved security by encoding NULL values; double free mitigation on by default; more precise free list corruption detection 2019-11-21 15:21:23 -08:00
daan
1674d551ff add verbose message with secure build level 2019-11-20 20:46:13 -08:00
Daan Leijen
049dbf41ba fix commit bits for huge page allocations 2019-11-20 15:44:07 -08:00
Daan Leijen
211f1aa519 remove reset delay slots; add reset tracking per page and segment 2019-11-20 14:55:12 -08:00
Daan Leijen
30e2c54adb remove delayed reset option (for now) 2019-11-20 14:13:02 -08:00
Daan Leijen
56b9fac4bf merge from dev 2019-11-20 13:31:04 -08:00
Daan Leijen
4d4a2885f5 use atomic read/write on the page->heap field where concurrent interaction is possible 2019-11-20 13:19:17 -08:00
daan
17ade3fca5 merge from dev 2019-11-13 17:52:20 -08:00
daan
a4ed63d127 Adresses pr #165 and issue #164 by @colesbury: 
On Mac OS, the thread-local _mi_default_heap may get reset before
_mi_thread_done is called, leaking the default heap on non-main threads.

Now the current default heap is also stored in mi_pthread_key (or mi_fls_key
on Windows). The _mi_thread_done function is called with this value.
 2019-11-13 17:22:03 -08:00
Daan Leijen
d01ed42bcb replace max_numa_nodes by use_numa_nodes (to help with wrong detection of numa nodes on WSL for example) 2019-11-13 13:35:50 -08:00
Daan Leijen
bdb8274819 change max_numa_node to max_numa_nodes option 2019-11-12 12:04:43 -08:00
daan
d4f54dcf30 remove numaif dependency on linux 2019-11-12 10:37:15 -08:00
daan
867d78f877 reserve huge OS pages earlier on at process_init 2019-11-12 10:19:52 -08:00
daan
af746ca4c1 inline bitmap_mask 2019-11-12 10:17:39 -08:00
daan
ef179a6377 avoid allocation at numa node detection on linux 2019-11-12 10:16:59 -08:00
daan
165ee45845 initialize delay slots for the main thread 2019-11-11 17:31:48 -08:00
daan
a0958b2da6 enable more reset delay slots 2019-11-11 17:06:16 -08:00
daan
5e6754f3f7 track commit status per block in a region 2019-11-11 15:45:31 -08:00
daan
db3f1c4bfa add commit info to arenas 2019-11-11 15:26:11 -08:00
daan
741a80256d merge from dev-exp 2019-11-11 14:51:16 -08:00
daan
2bb058bd25 remove cache_reset parameter 2019-11-11 14:44:32 -08:00
daan
534e1e39ef allow allocation in committed regions even if not requested 2019-11-11 14:42:29 -08:00
daan
93a6463383 only allow commit delay for small and medium objects 2019-11-11 14:16:45 -08:00
daan
83a066fd2d remove reset_decommits option 2019-11-11 09:46:02 -08:00
daan
21bbb1be87 fix warnings 2019-11-10 12:36:55 -08:00
daan
181bef382c merge dev-exp 2019-11-10 11:39:53 -08:00
Daan Leijen
9f08ddd0d0 refactor regions; add commit tracking on a segment basis 2019-11-09 19:30:53 -08:00
Daan Leijen
7b72a4cd50 fix region suitable bug 2019-11-08 11:55:43 -08:00
daan
6ccfb5a93e merge from dev 2019-11-07 11:00:50 -08:00
daan
9b65388807 fix space leak in secure mode where a non-null free list would be discarded 2019-11-07 10:59:19 -08:00
daan
27f1a8b3d2 fix avg display; set secure default to 0` 2019-11-07 10:35:30 -08:00
daan
36090dee5c Merge branch 'dev' into dev-exp 2019-11-07 10:34:06 -08:00
daan
31d11f64d5 fix secure free list extension where a non-empty initial free list was discarded 2019-11-07 10:33:45 -08:00
daan
378716c467 refactor and improve atomic bitmap usage 2019-11-07 10:26:52 -08:00
daan
b09282bc0d change arena allocator to atomic bitmap as well 2019-11-06 22:49:01 -08:00
daan
00e19cad9a refactor region code, split out atomic bitmap 2019-11-06 21:37:23 -08:00
daan
2887266063 optimize get numa node for single node systems 2019-11-06 14:17:36 -08:00
daan
829fd872f4 initial delay slots 2019-11-04 11:48:41 -08:00
daan
3d0a1e249f remove all floating point types and arithmetic 2019-11-04 09:40:10 -08:00
daan
c38af8f7c1 merge d1d65fbc: make max error messages configurable 2019-11-04 08:49:59 -08:00
daan
aece753dce fix output during preloading enabling stderr only after the crt has loaded 2019-11-04 08:46:17 -08:00
daan
8afd06b248 use int64 for time (instead of double) 2019-11-04 08:44:40 -08:00
daan
9d6a5acb22 fix unix build warnings 2019-11-03 13:34:54 -08:00
daan
d1d65fbca4 make max error messages configurable 2019-11-03 13:25:41 -08:00
daan
520a8dafee divide huge pages more even 2019-11-03 13:25:28 -08:00
daan
f36ec5d9d8 reserve huge pages incrementally 2019-11-03 13:16:07 -08:00
daan
e320488791 add numa nodes to stats 2019-11-03 12:18:32 -08:00
daan
723fbba259 fix output during preloading enabling stderr only after the crt has loaded 2019-11-03 12:18:20 -08:00
daan
62cd0237fc fix aligned huge page allocation on windows 2019-11-02 17:49:34 -07:00
daan
ee323aabac fix vs2017 build 2019-11-02 15:56:21 -07:00
daan
fd9d8c85ae change numa support on linux to use getcpu 2019-11-02 11:55:03 -07:00
daan
70748ee1ee fix missing semi colon 2019-11-02 10:39:09 -07:00
daan
a69016c33e improve and document numa support 2019-11-02 10:30:16 -07:00
daan
2c12d7f223 optimized numa calls; better Linux support 2019-11-01 22:01:52 -07:00
daan
57dd69265a normalize numa node 2019-11-01 20:30:01 -07:00
daan
2d10c78587 fix linux compilation 2019-11-01 20:19:00 -07:00
daan
3fadf4abaf initial numa awareness for regions 2019-11-01 20:01:08 -07:00
daan
a6499be074 initial numa support for arenas 2019-11-01 19:53:07 -07:00
daan
aaf01620f4 improve allocation of the huge OS page arena 2019-10-31 19:39:49 -07:00
daan
d36d04b4a6 add arena for huge page management 2019-10-31 15:35:10 -07:00
daan
8725a88fba fix assertion in debug secure mode 2019-10-30 15:22:40 -07:00
daan
4a4d74927c protect against double-free in multi-threaded free list 2019-10-30 14:53:21 -07:00
daan
6cf16b1201 fix reset error on windows when disabling eager commit option 2019-10-30 14:32:28 -07:00
daan
b052d3b731 enable double free and heap corruption detection in debug mode 2019-10-28 15:54:33 -07:00
daan
87bdfbb9b6 use more conservative retire strategy 2019-10-28 14:12:06 -07:00
daan
081e2d1eb6 fix statistics display 2019-10-28 13:43:42 -07:00
daan
28d4ec4c5a fix statistics accounting of huge pages 2019-10-28 13:14:14 -07:00
daan
5dfdc092b5 improve windows warning message 2019-10-28 12:26:57 -07:00
daan
25246070ae fix double free check in secure = 4 mode; inline _mi_ptr_cookie 2019-10-19 08:34:18 -07:00
daan
2affdbbd2e stronger secure mode when defining MI_SECURE=4: checks for double free, corrupted free list, and invalid pointer frees. Performance is impacted but not too much -- more perf testing is needed 2019-10-18 18:11:04 -07:00
daan
fdfa6ed260 fix warnings at high warning level in msvc 2019-10-17 16:56:57 -07:00
daan
5de851a84d update page_flags to have more portable definition 2019-10-17 16:48:16 -07:00
daan
93b4281b82 ensure randomized huge page start address in 1GiB aligned 2019-10-17 12:35:35 -07:00
daan
454bc67338 increase delayed output buffer to 32k 2019-10-17 12:14:15 -07:00
daan
26c27fbf58 use uint8_t bit fields, and improve portability of page_flags type 2019-10-17 12:07:26 -07:00
Johannes Schindelin
0fd0122c0a Avoid compiler warning when casting the result of GetProcAddress() 
It is most unfortunate that the return type of `GetProcAddress()` is
`FARPROC` (which is essentially `intptr_t(*)(void)): this type cannot
be cast by GCC without warnings to anything but the generic function
pointer type `void(*)(void)`.

Let's work around that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
 2019-10-17 20:43:18 +02:00
Johannes Schindelin
5bd8ea2e4f Repeat mi_decl_allocator in functions' definitions 
Quite a few functions are declared with that attribute, and VS 2019
complains if the definition does not repeat it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
 2019-10-17 20:43:18 +02:00
daan
08d83cc33d disallow regular allocation from the huge reserved area 2019-10-17 09:24:57 -07:00
daan
a96c90db5d remove old windows overriding method 2019-10-17 09:22:23 -07:00
daan
480f7683a9 cleanup delayed output 2019-10-12 10:42:04 -07:00
Daan Leijen
5e9b37dc4e add delayed output buffer 2019-10-12 10:21:42 -07:00
daan
1de6c22b49 remove incorrect region_count comment 2019-09-17 18:07:32 -07:00
daan
5744f90882 remove thread init/done verbose messages 2019-09-17 17:35:20 -07:00
daan
60efb62155 make secure a build option only 2019-09-17 17:31:42 -07:00
daan
d72b5350e3 fix error code on mprotect failure 2019-09-17 17:31:25 -07:00
daan
adf8e30eda ensure large page privileges are acquired on windows when calling reserve_huge_os_pages 2019-09-14 15:23:28 -07:00
daan
24777f6a91 limit aligned allocation to power-of-two alignment 2019-09-11 17:49:28 -07:00
daan
050e7cedf4 roll back commit 0aec6d93 on alignment >= size as it breaks assertions in tensorflow 2019-09-11 15:48:44 -07:00
Daan Leijen
1909cfb346 refine mi_os_good_alloc_size and use it for huge pages (to ensure realloc is bounded as in #153 2019-09-10 13:26:51 -07:00
Daan Leijen
d278c26c0e round huge page sizes into 12.5% increments (issue #153) 2019-09-10 07:54:31 -07:00
daan
ce81af1119 use mi_is_power_of_two when possible (pr #118) 2019-09-09 08:12:50 -07:00
daan
b104e434e4 fix volatile declaration on function 2019-09-09 08:07:35 -07:00
daan
23155c5d71 add recursion guard to deferred callback 2019-09-09 08:02:41 -07:00
daan
e227341f5b fix mi_collect being called with empty heap (issue #150) 2019-09-09 07:54:36 -07:00
daan
7b63bc4211 add alignment check to aligned_alloc for C11 compliance 2019-09-07 12:02:22 -07:00
daan
0aec6d9302 don't align to sizes greater than than the allocation size 2019-09-07 12:01:54 -07:00
daan
b241910810 don't allocate more than PTRDIFF_MAX on aligned allocations 2019-09-07 11:52:21 -07:00
daan
f2f7c8d0e9 fix volatile declaration on function pointer 2019-09-07 09:18:29 -07:00
daan
89858ab791 improve page_free_extend performance slightly; do not maintain expensive statistics in release mode 2019-09-06 16:48:48 -07:00