Commit Graph

556 Commits

Author SHA1 Message Date
daan
9b65388807 fix space leak in secure mode where a non-null free list would be discarded 2019-11-07 10:59:19 -08:00
daan
27f1a8b3d2 fix avg display; set secure default to 0` 2019-11-07 10:35:30 -08:00
daan
36090dee5c Merge branch 'dev' into dev-exp 2019-11-07 10:34:06 -08:00
daan
31d11f64d5 fix secure free list extension where a non-empty initial free list was discarded 2019-11-07 10:33:45 -08:00
daan
378716c467 refactor and improve atomic bitmap usage 2019-11-07 10:26:52 -08:00
daan
b09282bc0d change arena allocator to atomic bitmap as well 2019-11-06 22:49:01 -08:00
daan
00e19cad9a refactor region code, split out atomic bitmap 2019-11-06 21:37:23 -08:00
daan
2887266063 optimize get numa node for single node systems 2019-11-06 14:17:36 -08:00
daan
829fd872f4 initial delay slots 2019-11-04 11:48:41 -08:00
daan
3d0a1e249f remove all floating point types and arithmetic 2019-11-04 09:40:10 -08:00
daan
c38af8f7c1 merge d1d65fbc: make max error messages configurable 2019-11-04 08:49:59 -08:00
daan
aece753dce fix output during preloading enabling stderr only after the crt has loaded 2019-11-04 08:46:17 -08:00
daan
8afd06b248 use int64 for time (instead of double) 2019-11-04 08:44:40 -08:00
daan
9d6a5acb22 fix unix build warnings 2019-11-03 13:34:54 -08:00
daan
d1d65fbca4 make max error messages configurable 2019-11-03 13:25:41 -08:00
daan
520a8dafee divide huge pages more even 2019-11-03 13:25:28 -08:00
daan
f36ec5d9d8 reserve huge pages incrementally 2019-11-03 13:16:07 -08:00
daan
e320488791 add numa nodes to stats 2019-11-03 12:18:32 -08:00
daan
723fbba259 fix output during preloading enabling stderr only after the crt has loaded 2019-11-03 12:18:20 -08:00
daan
62cd0237fc fix aligned huge page allocation on windows 2019-11-02 17:49:34 -07:00
daan
ee323aabac fix vs2017 build 2019-11-02 15:56:21 -07:00
daan
fd9d8c85ae change numa support on linux to use getcpu 2019-11-02 11:55:03 -07:00
daan
70748ee1ee fix missing semi colon 2019-11-02 10:39:09 -07:00
daan
a69016c33e improve and document numa support 2019-11-02 10:30:16 -07:00
daan
2c12d7f223 optimized numa calls; better Linux support 2019-11-01 22:01:52 -07:00
daan
57dd69265a normalize numa node 2019-11-01 20:30:01 -07:00
daan
2d10c78587 fix linux compilation 2019-11-01 20:19:00 -07:00
daan
3fadf4abaf initial numa awareness for regions 2019-11-01 20:01:08 -07:00
daan
a6499be074 initial numa support for arenas 2019-11-01 19:53:07 -07:00
daan
aaf01620f4 improve allocation of the huge OS page arena 2019-10-31 19:39:49 -07:00
daan
d36d04b4a6 add arena for huge page management 2019-10-31 15:35:10 -07:00
daan
8725a88fba fix assertion in debug secure mode 2019-10-30 15:22:40 -07:00
daan
4a4d74927c protect against double-free in multi-threaded free list 2019-10-30 14:53:21 -07:00
daan
6cf16b1201 fix reset error on windows when disabling eager commit option 2019-10-30 14:32:28 -07:00
daan
b052d3b731 enable double free and heap corruption detection in debug mode 2019-10-28 15:54:33 -07:00
daan
87bdfbb9b6 use more conservative retire strategy 2019-10-28 14:12:06 -07:00
daan
081e2d1eb6 fix statistics display 2019-10-28 13:43:42 -07:00
daan
28d4ec4c5a fix statistics accounting of huge pages 2019-10-28 13:14:14 -07:00
daan
5dfdc092b5 improve windows warning message 2019-10-28 12:26:57 -07:00
daan
25246070ae fix double free check in secure = 4 mode; inline _mi_ptr_cookie 2019-10-19 08:34:18 -07:00
daan
2affdbbd2e stronger secure mode when defining MI_SECURE=4: checks for double free, corrupted free list, and invalid pointer frees. Performance is impacted but not too much -- more perf testing is needed 2019-10-18 18:11:04 -07:00
daan
fdfa6ed260 fix warnings at high warning level in msvc 2019-10-17 16:56:57 -07:00
daan
5de851a84d update page_flags to have more portable definition 2019-10-17 16:48:16 -07:00
daan
93b4281b82 ensure randomized huge page start address in 1GiB aligned 2019-10-17 12:35:35 -07:00
daan
454bc67338 increase delayed output buffer to 32k 2019-10-17 12:14:15 -07:00
daan
26c27fbf58 use uint8_t bit fields, and improve portability of page_flags type 2019-10-17 12:07:26 -07:00
Johannes Schindelin
0fd0122c0a Avoid compiler warning when casting the result of GetProcAddress()
It is most unfortunate that the return type of `GetProcAddress()` is
`FARPROC` (which is essentially `intptr_t(*)(void)): this type cannot
be cast by GCC without warnings to anything but the generic function
pointer type `void(*)(void)`.

Let's work around that.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2019-10-17 20:43:18 +02:00
Johannes Schindelin
5bd8ea2e4f Repeat mi_decl_allocator in functions' definitions
Quite a few functions are declared with that attribute, and VS 2019
complains if the definition does not repeat it.

Signed-off-by: Johannes Schindelin <johannes.schindelin@gmx.de>
2019-10-17 20:43:18 +02:00
daan
08d83cc33d disallow regular allocation from the huge reserved area 2019-10-17 09:24:57 -07:00
daan
a96c90db5d remove old windows overriding method 2019-10-17 09:22:23 -07:00
daan
480f7683a9 cleanup delayed output 2019-10-12 10:42:04 -07:00
Daan Leijen
5e9b37dc4e add delayed output buffer 2019-10-12 10:21:42 -07:00
daan
1de6c22b49 remove incorrect region_count comment 2019-09-17 18:07:32 -07:00
daan
5744f90882 remove thread init/done verbose messages 2019-09-17 17:35:20 -07:00
daan
60efb62155 make secure a build option only 2019-09-17 17:31:42 -07:00
daan
d72b5350e3 fix error code on mprotect failure 2019-09-17 17:31:25 -07:00
daan
adf8e30eda ensure large page privileges are acquired on windows when calling reserve_huge_os_pages 2019-09-14 15:23:28 -07:00
daan
24777f6a91 limit aligned allocation to power-of-two alignment 2019-09-11 17:49:28 -07:00
daan
050e7cedf4 roll back commit 0aec6d93 on alignment >= size as it breaks assertions in tensorflow 2019-09-11 15:48:44 -07:00
Daan Leijen
1909cfb346 refine mi_os_good_alloc_size and use it for huge pages (to ensure realloc is bounded as in #153 2019-09-10 13:26:51 -07:00
Daan Leijen
d278c26c0e round huge page sizes into 12.5% increments (issue #153) 2019-09-10 07:54:31 -07:00
daan
ce81af1119 use mi_is_power_of_two when possible (pr #118) 2019-09-09 08:12:50 -07:00
daan
b104e434e4 fix volatile declaration on function 2019-09-09 08:07:35 -07:00
daan
23155c5d71 add recursion guard to deferred callback 2019-09-09 08:02:41 -07:00
daan
e227341f5b fix mi_collect being called with empty heap (issue #150) 2019-09-09 07:54:36 -07:00
daan
7b63bc4211 add alignment check to aligned_alloc for C11 compliance 2019-09-07 12:02:22 -07:00
daan
0aec6d9302 don't align to sizes greater than than the allocation size 2019-09-07 12:01:54 -07:00
daan
b241910810 don't allocate more than PTRDIFF_MAX on aligned allocations 2019-09-07 11:52:21 -07:00
daan
f2f7c8d0e9 fix volatile declaration on function pointer 2019-09-07 09:18:29 -07:00
daan
89858ab791 improve page_free_extend performance slightly; do not maintain expensive statistics in release mode 2019-09-06 16:48:48 -07:00
daan
47fc73d694 fix c++ compilation; make default output non-atomic 2019-09-06 11:51:22 -07:00
daan
f38fcf79eb ensure addresses of large aligned allocations are randomized 2019-09-06 09:19:26 -07:00
Jakub Szymanski
1a7b1783fa changed strategy to reduce contention for region search and vs2019 project fix 2019-09-05 08:13:52 -07:00
daan
2be3208734 add wrap-around for large aligned allocations on Windows and Linux 2019-09-04 21:03:00 -07:00
daan
e73982c4cf better alignment for large and huge pages inside a segment 2019-09-04 18:42:37 -07:00
daan
3bbc047ba6 reserve huge pages returns actual number of pages reserved 2019-09-04 12:26:02 -07:00
daan
6c43ae2bde add later thread-done invokation; add mi_is_redirected call 2019-09-04 12:00:41 -07:00
daan
3c563816c0 read regions count on every loop iteration 2019-09-03 18:25:01 -07:00
daan
aea21486dd fix regions_count tracking in case of concurrent region allocations 2019-09-03 18:20:56 -07:00
daan
a6a956c589 add back zero initialized re-allocation 2019-09-03 10:57:39 -07:00
daan
7a91368108 pass page to block_zero_init and extend to block_size 2019-09-03 10:11:24 -07:00
daan
739d11313c fix commit zero on windows 2019-09-02 14:53:22 -07:00
daan
d1bd1644d5 support zero-initialized memory detection 2019-09-02 13:16:52 -07:00
daan
31726c6554 fix type; fix comment 2019-09-02 10:18:34 -07:00
daan
d6e35ffd83 Merge branch 'dev' into dev-exp 2019-09-02 10:10:08 -07:00
daan
783ae72743 Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev 2019-09-02 09:53:30 -07:00
daan
e874844562 update redirection to include fixes for x86 2019-09-01 12:18:29 -07:00
daan
ab5af8fdad merge from dev 2019-08-30 14:38:28 -07:00
daan
9fc726619a guard against passing stdin/stderr as output 2019-08-30 14:37:04 -07:00
daan
86ee1722f5 in verbose mode, list all options at startup 2019-08-30 14:34:46 -07:00
daan
f35f643508 merge from dev 2019-08-29 09:52:22 -07:00
daan
7bf12c7b5f make output function configurable; remove <stdio.h> from standard includes 2019-08-29 09:42:50 -07:00
daan
4819d3f78f expose mi_stats_merge function 2019-08-29 09:01:00 -07:00
daan
4b39c0b06e fix eager_delay test, and allow large OS pages even without eager commit enabled 2019-08-29 07:55:57 -07:00
daan
64c1d6de86 fix mi_likely branch that was marked as unlikely 2019-08-29 07:50:35 -07:00
daan
e8c7505853 only set has_aligned flag if really necessary 2019-08-29 07:49:40 -07:00
daan
154fd471a1 fix comparison warning 2019-08-29 07:48:15 -07:00
daan
d381fcd9fa rename lazy to eager_commit_delay 2019-08-28 12:09:23 -07:00
daan
9af51506a6 track commit and is_large status more precisely 2019-08-28 11:58:30 -07:00
David Carlier
3903a10e93 FreeBSD (and possibly NetBSD) build fix. 2019-08-28 07:17:58 +00:00
daan
a551f3abc4 more precise commit statistics 2019-08-27 18:08:03 -07:00
daan
18e02c3766 try allocating non-eager segments in non-fixed memory 2019-08-27 17:02:56 -07:00
daan
23b5593af6 Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev 2019-08-27 16:40:47 -07:00
David Carlier
b72a2d9659 macOS: anonymous page ID make it as env var. 2019-08-27 19:17:41 +01:00
daan
db8d443ae6 track more precisely if memory is fixed or committed 2019-08-26 22:45:26 -07:00
daan
eea093000a graceful fallback for huge page allocation on Linux 2019-08-26 13:47:52 -07:00
daan
3d8c331a1c search regions always from the lowest index 2019-08-26 12:41:59 -07:00
daan
8b06ab1e49 fix check on gigabyte alignment of huge os pages on windows 2019-08-26 12:41:35 -07:00
daan
a654732d99 merge from dev new atomics 2019-08-26 12:30:03 -07:00
daan
2c19388bcf initialize mimalloc options at process load 2019-08-26 11:44:41 -07:00
daan
7ce9c02fd4 make cas weak use release memory order; improve free assembly 2019-08-26 08:20:26 -07:00
daan
5c7c106d62 strengthen some atomic operations for weak memory models 2019-08-26 08:11:58 -07:00
daan
2159c22415 fix atomic declaration on windows 2019-08-25 23:06:18 -07:00
daan
baabc77503 use proper atomic initialization macros 2019-08-25 23:02:41 -07:00
daan
e8664001f7 Use standard _Atomic declarations and clean up atomic operations 2019-08-25 22:59:12 -07:00
daan
6e360d34ee fix 1GB huge page flag on Linux 2019-08-25 13:15:26 -07:00
daan
23c35f4aba merge from dev 2019-08-25 13:07:07 -07:00
daan
b86c851cca put segment cache behind an option and disable by default 2019-08-25 12:35:14 -07:00
daan
038e8fd7d6 reduce retire size 2019-08-25 12:10:18 -07:00
daan
c3c6401633 merge from master; fix windows environment reading 2019-08-25 11:36:34 -07:00
daan
7972b64a82 resolve race on option initialization 2019-08-25 11:18:16 -07:00
daan
a431d80fc3 better object sizes for large objects 2019-08-25 10:20:03 -07:00
daan
23812cc0ac do not keep a queue of huge pages and free them directly 2019-08-24 15:45:14 -07:00
daan
d04d379f66 fix merge conflicts 2019-08-23 21:44:07 -07:00
daan
d3224d0bba Merge branch 'dev' into dev-win 2019-08-23 21:42:32 -07:00
daan
6c6fcad242 remove threadid from pages and keep page flags separate (cherry picked) 2019-08-23 21:42:24 -07:00
daan
15552eba79 ensure volatile declaration for abandoned_next field 2019-08-23 21:39:27 -07:00
daan
25ea9cf142 on windows use 4TiB area for aligned allocation 2019-08-23 21:38:45 -07:00
daan
acde83543f remove threadid from pages and keep page flags separate 2019-08-23 14:08:00 -07:00
daan
b511309709 ensure volatile declaration for abandoned_next field 2019-08-23 13:44:43 -07:00
daan
a1c5218ff5 use 4TiB area on windows 64-bit for aligned allocation 2019-08-23 11:33:06 -07:00
daan
5d3bf1c844 don't commit or reset in huge OS pages 2019-08-23 11:22:35 -07:00
daan
71121a7457 Merge branch 'dev-win' into dev-exp 2019-08-21 11:12:25 -07:00
daan
e7bd8c5d14 merge with dev-win 2019-08-21 11:11:40 -07:00
Jakub Szymanski
ee475fd8cd add warning when no avaiable mem 2019-08-21 11:11:36 -07:00
Daan Leijen
13ca6a088b Merge branch 'dev' into dev-win 2019-08-21 08:11:26 -07:00
Daan Leijen
311794b296 remove debug statement 2019-08-21 08:11:16 -07:00
Daan Leijen
12257e5bc1 fix thread-id reset on page abandonment 2019-08-21 08:09:28 -07:00
Daan Leijen
cc6d14c55b fix thread-id reset on page abandonment 2019-08-21 08:06:40 -07:00
daan
b005f7cdbf Merge branch 'dev' into dev-win 2019-08-20 09:46:05 -07:00
daan
ff88361329 lower block size for keeping retired pages 2019-08-20 09:45:50 -07:00
daan
d81b800e12 optimize stat counter increase 2019-08-20 08:58:53 -07:00
daan
86d4ded831 Merge branch 'dev' into dev-exp 2019-08-20 08:27:46 -07:00
daan
a5071eb510 Merge branch 'dev' into dev-win 2019-08-20 07:50:03 -07:00
daan
03d994648a better option interface: option_enable, option_disable, and option_set_enabled 2019-08-20 07:47:50 -07:00
daan
7a2e0df5ea merge from dev 2019-08-20 07:37:46 -07:00
daan
1e0cd575a5 do not export DllEntry on windows 2019-08-20 07:06:53 -07:00
daan
d52e4039b6 remove the reset_discards option 2019-08-20 07:06:11 -07:00
daan
79487dbedf add checks for right option order 2019-08-20 06:58:51 -07:00
daan
29e73dec22 Merge branch 'dev' into dev-win 2019-08-19 21:25:09 -07:00
daan
8c795cc37d remove test statement 2019-08-19 21:24:38 -07:00
daan
05631ebfc4 Merge branch 'dev' into dev-win 2019-08-19 21:23:04 -07:00
daan
d2324f1c2a always check large_os_pages enabled option so it can be changed during the running of the program 2019-08-19 21:22:24 -07:00
daan
48a7d64bdd if region overcommitting due to contention save it to a further slot 2019-08-19 21:20:24 -07:00
daan
2066e2797d don't allow allocations larger than PTRDIFF_MAX, issue #121 2019-08-19 19:29:27 -07:00
daan
74299646bb fix heap cookie and random order on initialization (issue #131) 2019-08-19 19:22:13 -07:00
daan
b46bacc180 fix absorb heap (issue #132) 2019-08-19 19:19:09 -07:00
daan
82e1b40166 Merge branch 'master' into dev 2019-08-19 19:13:25 -07:00
daan
a8b24472cb use 1GiB huge pages on windows when reserving upfront 2019-08-19 18:16:12 -07:00
daan
0e639addb0 add environment option to reserve huge pages upfront 2019-08-19 14:14:50 -07:00
daan
5b2f4184b3 Merge branch 'dev' into dev-exp 2019-08-19 11:37:39 -07:00
daan
741f37e1f0 always check large_os_pages enabled option so it can be changed during the running of the program 2019-08-19 11:17:00 -07:00
daan
42dedb00ec initial implementation of mi_os_reserve_huge_pages 2019-08-19 11:10:06 -07:00
daan
61ebaaa403 Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev 2019-08-19 08:10:11 -07:00
Daan Leijen
367fb046ec Merge branch 'dev' into dev-win 2019-08-16 18:40:56 -07:00
Daan Leijen
069c26dbab fix small object size check in mi_segment_page_alloc 2019-08-16 18:40:30 -07:00
daan
6d11e59250 fix to avoid potential linear behavior in page collect 2019-08-14 07:46:38 -07:00
David Carlier
b0003cb101 linux build correction
MAP_HUGE_2MB is linux specific and the related header
is not automatically included.
2019-08-14 15:07:13 +01:00
daan
d3375f2bac fix assertion in mi_free_size 2019-08-12 13:50:41 -07:00
daan
ab53da679e Merge branch 'dev' into dev-exp 2019-08-12 13:43:29 -07:00
daan
178ea0840c roll back commit #72d8608 to avoid spin lock contention on region allocation 2019-08-12 13:40:19 -07:00
daan
113a3cc1a2 Merge branch 'dev' into dev-exp 2019-08-12 09:20:16 -07:00
daan
8ca79b5359 Hide MEM_EXTENDED_PARAMETER to compile on older SDK's 2019-08-12 08:58:22 -07:00
daan
8b2194c160 merge from master 2019-08-12 15:50:21 -07:00
JeanMiK
3f43cbe6a1 Fix C++ signature of function mi_is_in_heap_region
Original code did not compile with MI_USE_CXX=ON as function
mi_is_in_heap_region was declared noexcept through macro mi_attr_noexcept
but defined without specification, i.e. implicitly with noexception(false)

Just adding the macro in the definition is enough to fix the issue
(tested with gcc 8.3.5)
2019-08-12 12:43:01 +02:00
daan
63d4d2ad9c Merge branch 'dev' into dev-exp 2019-08-12 01:59:31 -07:00
daan
648f535c80 improve support for transparent huge pages on Linux (pr #115) 2019-08-11 18:23:31 -07:00
daan
f35c2c5201 improve message printing from redirection module 2019-08-11 16:38:58 -07:00
daan
0c912445c4 Merge branch 'dev' into dev-exp 2019-08-11 14:46:53 -07:00
daan
feb7b2c160 improve bit range skipping in region search 2019-08-11 13:49:12 -07:00
daan
3a75a9d5be use bit-scan instruction to speed up region search 2019-08-11 13:03:00 -07:00
daan
0fd898315c per thread region search index 2019-08-11 12:15:13 -07:00
daan
2e924150ae further layout improvement for msvc code generation 2019-08-11 11:30:24 -07:00
daan
7b105c4810 improve layout of page for x64 2019-08-11 10:31:00 -07:00
daan
2c1da9d194 fix scalar initialization of page flags 2019-08-11 09:03:52 -07:00
daan
d71e67b689 Merge branch 'dev' into dev-exp 2019-08-11 09:00:39 -07:00
daan
3a171624bd Merge branch 'master' into dev 2019-08-11 08:59:38 -07:00
daan
a73daf1804 avoid mod operation in region allocation and limit extension 2019-08-11 08:54:32 -07:00
daan
756f4b2724 improve bit scanning code in region allocation 2019-08-11 08:48:19 -07:00
daan
6435887657 add comment on wait strategy in region allocation 2019-08-11 08:18:51 -07:00
daan
f5902a7a82 ensure webassembly allocation sizes are page aligned 2019-08-11 08:06:17 -07:00
daan
a1384bff72 avoid atomic read on aligned allocation on Linux 2019-08-11 08:00:10 -07:00
daan
dbe010f2d4 add huge and giant stats; increase bins 2019-08-10 20:51:37 -07:00
daan
737b05cc25 remove unused fields from OS thread local data 2019-08-10 17:50:27 -07:00
daan
ed785253bf restructure the page flags to use explicit masks 2019-08-10 17:48:00 -07:00
daan
74a5c17178 update order of options; make eager_region_commit 0 on windows by default 2019-08-10 15:44:55 -07:00
daan
2fee6f98d7 Merge branch 'master' into dev 2019-08-10 15:23:43 -07:00
daan
2b9e3ed01b merge from dev-win 2019-08-10 15:19:04 -07:00
daan
78a70ada2e Fix option reading even when preloading on Windows 2019-08-10 15:17:40 -07:00
daan
d9cbc457b8 Merge branch 'dev-win' into dev-exp 2019-08-10 15:11:51 -07:00