daan
60efb62155
make secure a build option only
2019-09-17 17:31:42 -07:00
daan
d72b5350e3
fix error code on mprotect failure
2019-09-17 17:31:25 -07:00
daan
adf8e30eda
ensure large page privileges are acquired on windows when calling reserve_huge_os_pages
2019-09-14 15:23:28 -07:00
daan
24777f6a91
limit aligned allocation to power-of-two alignment
2019-09-11 17:49:28 -07:00
daan
050e7cedf4
roll back commit 0aec6d93 on alignment >= size as it breaks assertions in tensorflow
2019-09-11 15:48:44 -07:00
Daan Leijen
1909cfb346
refine mi_os_good_alloc_size and use it for huge pages (to ensure realloc is bounded as in #153
2019-09-10 13:26:51 -07:00
Daan Leijen
d278c26c0e
round huge page sizes into 12.5% increments (issue #153 )
2019-09-10 07:54:31 -07:00
daan
ce81af1119
use mi_is_power_of_two when possible (pr #118 )
2019-09-09 08:12:50 -07:00
daan
b104e434e4
fix volatile declaration on function
2019-09-09 08:07:35 -07:00
daan
23155c5d71
add recursion guard to deferred callback
2019-09-09 08:02:41 -07:00
daan
e227341f5b
fix mi_collect being called with empty heap (issue #150 )
2019-09-09 07:54:36 -07:00
daan
7b63bc4211
add alignment check to aligned_alloc for C11 compliance
2019-09-07 12:02:22 -07:00
daan
0aec6d9302
don't align to sizes greater than than the allocation size
2019-09-07 12:01:54 -07:00
daan
b241910810
don't allocate more than PTRDIFF_MAX on aligned allocations
2019-09-07 11:52:21 -07:00
daan
f2f7c8d0e9
fix volatile declaration on function pointer
2019-09-07 09:18:29 -07:00
daan
89858ab791
improve page_free_extend performance slightly; do not maintain expensive statistics in release mode
2019-09-06 16:48:48 -07:00
daan
47fc73d694
fix c++ compilation; make default output non-atomic
2019-09-06 11:51:22 -07:00
daan
f38fcf79eb
ensure addresses of large aligned allocations are randomized
2019-09-06 09:19:26 -07:00
Jakub Szymanski
1a7b1783fa
changed strategy to reduce contention for region search and vs2019 project fix
2019-09-05 08:13:52 -07:00
daan
2be3208734
add wrap-around for large aligned allocations on Windows and Linux
2019-09-04 21:03:00 -07:00
daan
e73982c4cf
better alignment for large and huge pages inside a segment
2019-09-04 18:42:37 -07:00
daan
3bbc047ba6
reserve huge pages returns actual number of pages reserved
2019-09-04 12:26:02 -07:00
daan
6c43ae2bde
add later thread-done invokation; add mi_is_redirected call
2019-09-04 12:00:41 -07:00
daan
3c563816c0
read regions count on every loop iteration
2019-09-03 18:25:01 -07:00
daan
aea21486dd
fix regions_count tracking in case of concurrent region allocations
2019-09-03 18:20:56 -07:00
daan
a6a956c589
add back zero initialized re-allocation
2019-09-03 10:57:39 -07:00
daan
7a91368108
pass page to block_zero_init and extend to block_size
2019-09-03 10:11:24 -07:00
daan
739d11313c
fix commit zero on windows
2019-09-02 14:53:22 -07:00
daan
d1bd1644d5
support zero-initialized memory detection
2019-09-02 13:16:52 -07:00
daan
31726c6554
fix type; fix comment
2019-09-02 10:18:34 -07:00
daan
d6e35ffd83
Merge branch 'dev' into dev-exp
2019-09-02 10:10:08 -07:00
daan
783ae72743
Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev
2019-09-02 09:53:30 -07:00
daan
e874844562
update redirection to include fixes for x86
2019-09-01 12:18:29 -07:00
daan
ab5af8fdad
merge from dev
2019-08-30 14:38:28 -07:00
daan
9fc726619a
guard against passing stdin/stderr as output
2019-08-30 14:37:04 -07:00
daan
86ee1722f5
in verbose mode, list all options at startup
2019-08-30 14:34:46 -07:00
daan
f35f643508
merge from dev
2019-08-29 09:52:22 -07:00
daan
7bf12c7b5f
make output function configurable; remove <stdio.h> from standard includes
2019-08-29 09:42:50 -07:00
daan
4819d3f78f
expose mi_stats_merge function
2019-08-29 09:01:00 -07:00
daan
4b39c0b06e
fix eager_delay test, and allow large OS pages even without eager commit enabled
2019-08-29 07:55:57 -07:00
daan
64c1d6de86
fix mi_likely branch that was marked as unlikely
2019-08-29 07:50:35 -07:00
daan
e8c7505853
only set has_aligned flag if really necessary
2019-08-29 07:49:40 -07:00
daan
154fd471a1
fix comparison warning
2019-08-29 07:48:15 -07:00
daan
d381fcd9fa
rename lazy to eager_commit_delay
2019-08-28 12:09:23 -07:00
daan
9af51506a6
track commit and is_large status more precisely
2019-08-28 11:58:30 -07:00
David Carlier
3903a10e93
FreeBSD (and possibly NetBSD) build fix.
2019-08-28 07:17:58 +00:00
daan
a551f3abc4
more precise commit statistics
2019-08-27 18:08:03 -07:00
daan
18e02c3766
try allocating non-eager segments in non-fixed memory
2019-08-27 17:02:56 -07:00
daan
23b5593af6
Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev
2019-08-27 16:40:47 -07:00
David Carlier
b72a2d9659
macOS: anonymous page ID make it as env var.
2019-08-27 19:17:41 +01:00
daan
db8d443ae6
track more precisely if memory is fixed or committed
2019-08-26 22:45:26 -07:00
daan
eea093000a
graceful fallback for huge page allocation on Linux
2019-08-26 13:47:52 -07:00
daan
3d8c331a1c
search regions always from the lowest index
2019-08-26 12:41:59 -07:00
daan
8b06ab1e49
fix check on gigabyte alignment of huge os pages on windows
2019-08-26 12:41:35 -07:00
daan
a654732d99
merge from dev new atomics
2019-08-26 12:30:03 -07:00
daan
2c19388bcf
initialize mimalloc options at process load
2019-08-26 11:44:41 -07:00
daan
7ce9c02fd4
make cas weak use release memory order; improve free assembly
2019-08-26 08:20:26 -07:00
daan
5c7c106d62
strengthen some atomic operations for weak memory models
2019-08-26 08:11:58 -07:00
daan
2159c22415
fix atomic declaration on windows
2019-08-25 23:06:18 -07:00
daan
baabc77503
use proper atomic initialization macros
2019-08-25 23:02:41 -07:00
daan
e8664001f7
Use standard _Atomic declarations and clean up atomic operations
2019-08-25 22:59:12 -07:00
daan
6e360d34ee
fix 1GB huge page flag on Linux
2019-08-25 13:15:26 -07:00
daan
23c35f4aba
merge from dev
2019-08-25 13:07:07 -07:00
daan
b86c851cca
put segment cache behind an option and disable by default
2019-08-25 12:35:14 -07:00
daan
038e8fd7d6
reduce retire size
2019-08-25 12:10:18 -07:00
daan
c3c6401633
merge from master; fix windows environment reading
2019-08-25 11:36:34 -07:00
daan
7972b64a82
resolve race on option initialization
2019-08-25 11:18:16 -07:00
daan
a431d80fc3
better object sizes for large objects
2019-08-25 10:20:03 -07:00
daan
23812cc0ac
do not keep a queue of huge pages and free them directly
2019-08-24 15:45:14 -07:00
daan
d04d379f66
fix merge conflicts
2019-08-23 21:44:07 -07:00
daan
d3224d0bba
Merge branch 'dev' into dev-win
2019-08-23 21:42:32 -07:00
daan
6c6fcad242
remove threadid from pages and keep page flags separate (cherry picked)
2019-08-23 21:42:24 -07:00
daan
15552eba79
ensure volatile declaration for abandoned_next field
2019-08-23 21:39:27 -07:00
daan
25ea9cf142
on windows use 4TiB area for aligned allocation
2019-08-23 21:38:45 -07:00
daan
acde83543f
remove threadid from pages and keep page flags separate
2019-08-23 14:08:00 -07:00
daan
b511309709
ensure volatile declaration for abandoned_next field
2019-08-23 13:44:43 -07:00
daan
a1c5218ff5
use 4TiB area on windows 64-bit for aligned allocation
2019-08-23 11:33:06 -07:00
daan
5d3bf1c844
don't commit or reset in huge OS pages
2019-08-23 11:22:35 -07:00
daan
71121a7457
Merge branch 'dev-win' into dev-exp
2019-08-21 11:12:25 -07:00
daan
e7bd8c5d14
merge with dev-win
2019-08-21 11:11:40 -07:00
Jakub Szymanski
ee475fd8cd
add warning when no avaiable mem
2019-08-21 11:11:36 -07:00
Daan Leijen
13ca6a088b
Merge branch 'dev' into dev-win
2019-08-21 08:11:26 -07:00
Daan Leijen
311794b296
remove debug statement
2019-08-21 08:11:16 -07:00
Daan Leijen
12257e5bc1
fix thread-id reset on page abandonment
2019-08-21 08:09:28 -07:00
Daan Leijen
cc6d14c55b
fix thread-id reset on page abandonment
2019-08-21 08:06:40 -07:00
daan
b005f7cdbf
Merge branch 'dev' into dev-win
2019-08-20 09:46:05 -07:00
daan
ff88361329
lower block size for keeping retired pages
2019-08-20 09:45:50 -07:00
daan
d81b800e12
optimize stat counter increase
2019-08-20 08:58:53 -07:00
daan
86d4ded831
Merge branch 'dev' into dev-exp
2019-08-20 08:27:46 -07:00
daan
a5071eb510
Merge branch 'dev' into dev-win
2019-08-20 07:50:03 -07:00
daan
03d994648a
better option interface: option_enable, option_disable, and option_set_enabled
2019-08-20 07:47:50 -07:00
daan
7a2e0df5ea
merge from dev
2019-08-20 07:37:46 -07:00
daan
1e0cd575a5
do not export DllEntry on windows
2019-08-20 07:06:53 -07:00
daan
d52e4039b6
remove the reset_discards option
2019-08-20 07:06:11 -07:00
daan
79487dbedf
add checks for right option order
2019-08-20 06:58:51 -07:00
daan
29e73dec22
Merge branch 'dev' into dev-win
2019-08-19 21:25:09 -07:00
daan
8c795cc37d
remove test statement
2019-08-19 21:24:38 -07:00
daan
05631ebfc4
Merge branch 'dev' into dev-win
2019-08-19 21:23:04 -07:00
daan
d2324f1c2a
always check large_os_pages enabled option so it can be changed during the running of the program
2019-08-19 21:22:24 -07:00
daan
48a7d64bdd
if region overcommitting due to contention save it to a further slot
2019-08-19 21:20:24 -07:00
daan
2066e2797d
don't allow allocations larger than PTRDIFF_MAX, issue #121
2019-08-19 19:29:27 -07:00
daan
74299646bb
fix heap cookie and random order on initialization (issue #131 )
2019-08-19 19:22:13 -07:00
daan
b46bacc180
fix absorb heap (issue #132 )
2019-08-19 19:19:09 -07:00
daan
82e1b40166
Merge branch 'master' into dev
2019-08-19 19:13:25 -07:00
daan
a8b24472cb
use 1GiB huge pages on windows when reserving upfront
2019-08-19 18:16:12 -07:00
daan
0e639addb0
add environment option to reserve huge pages upfront
2019-08-19 14:14:50 -07:00
daan
5b2f4184b3
Merge branch 'dev' into dev-exp
2019-08-19 11:37:39 -07:00
daan
741f37e1f0
always check large_os_pages enabled option so it can be changed during the running of the program
2019-08-19 11:17:00 -07:00
daan
42dedb00ec
initial implementation of mi_os_reserve_huge_pages
2019-08-19 11:10:06 -07:00
daan
61ebaaa403
Merge branch 'dev' of https://github.com/microsoft/mimalloc into dev
2019-08-19 08:10:11 -07:00
Daan Leijen
367fb046ec
Merge branch 'dev' into dev-win
2019-08-16 18:40:56 -07:00
Daan Leijen
069c26dbab
fix small object size check in mi_segment_page_alloc
2019-08-16 18:40:30 -07:00
daan
6d11e59250
fix to avoid potential linear behavior in page collect
2019-08-14 07:46:38 -07:00
David Carlier
b0003cb101
linux build correction
...
MAP_HUGE_2MB is linux specific and the related header
is not automatically included.
2019-08-14 15:07:13 +01:00
daan
d3375f2bac
fix assertion in mi_free_size
2019-08-12 13:50:41 -07:00
daan
ab53da679e
Merge branch 'dev' into dev-exp
2019-08-12 13:43:29 -07:00
daan
178ea0840c
roll back commit #72d8608 to avoid spin lock contention on region allocation
2019-08-12 13:40:19 -07:00
daan
113a3cc1a2
Merge branch 'dev' into dev-exp
2019-08-12 09:20:16 -07:00
daan
8ca79b5359
Hide MEM_EXTENDED_PARAMETER to compile on older SDK's
2019-08-12 08:58:22 -07:00
daan
8b2194c160
merge from master
2019-08-12 15:50:21 -07:00
JeanMiK
3f43cbe6a1
Fix C++ signature of function mi_is_in_heap_region
...
Original code did not compile with MI_USE_CXX=ON as function
mi_is_in_heap_region was declared noexcept through macro mi_attr_noexcept
but defined without specification, i.e. implicitly with noexception(false)
Just adding the macro in the definition is enough to fix the issue
(tested with gcc 8.3.5)
2019-08-12 12:43:01 +02:00
daan
63d4d2ad9c
Merge branch 'dev' into dev-exp
2019-08-12 01:59:31 -07:00
daan
648f535c80
improve support for transparent huge pages on Linux (pr #115 )
2019-08-11 18:23:31 -07:00
daan
f35c2c5201
improve message printing from redirection module
2019-08-11 16:38:58 -07:00
daan
0c912445c4
Merge branch 'dev' into dev-exp
2019-08-11 14:46:53 -07:00
daan
feb7b2c160
improve bit range skipping in region search
2019-08-11 13:49:12 -07:00
daan
3a75a9d5be
use bit-scan instruction to speed up region search
2019-08-11 13:03:00 -07:00
daan
0fd898315c
per thread region search index
2019-08-11 12:15:13 -07:00
daan
2e924150ae
further layout improvement for msvc code generation
2019-08-11 11:30:24 -07:00
daan
7b105c4810
improve layout of page for x64
2019-08-11 10:31:00 -07:00
daan
2c1da9d194
fix scalar initialization of page flags
2019-08-11 09:03:52 -07:00
daan
d71e67b689
Merge branch 'dev' into dev-exp
2019-08-11 09:00:39 -07:00
daan
3a171624bd
Merge branch 'master' into dev
2019-08-11 08:59:38 -07:00
daan
a73daf1804
avoid mod operation in region allocation and limit extension
2019-08-11 08:54:32 -07:00
daan
756f4b2724
improve bit scanning code in region allocation
2019-08-11 08:48:19 -07:00
daan
6435887657
add comment on wait strategy in region allocation
2019-08-11 08:18:51 -07:00
daan
f5902a7a82
ensure webassembly allocation sizes are page aligned
2019-08-11 08:06:17 -07:00
daan
a1384bff72
avoid atomic read on aligned allocation on Linux
2019-08-11 08:00:10 -07:00
daan
dbe010f2d4
add huge and giant stats; increase bins
2019-08-10 20:51:37 -07:00
daan
737b05cc25
remove unused fields from OS thread local data
2019-08-10 17:50:27 -07:00
daan
ed785253bf
restructure the page flags to use explicit masks
2019-08-10 17:48:00 -07:00
daan
74a5c17178
update order of options; make eager_region_commit 0 on windows by default
2019-08-10 15:44:55 -07:00
daan
2fee6f98d7
Merge branch 'master' into dev
2019-08-10 15:23:43 -07:00
daan
2b9e3ed01b
merge from dev-win
2019-08-10 15:19:04 -07:00
daan
78a70ada2e
Fix option reading even when preloading on Windows
2019-08-10 15:17:40 -07:00
daan
d9cbc457b8
Merge branch 'dev-win' into dev-exp
2019-08-10 15:11:51 -07:00
daan
6f3bc87dcd
merge from dev
2019-08-10 10:26:46 -07:00
daan
8ae2492eee
fix arguments for the heap visitor function, issue #124
2019-08-10 10:14:01 -07:00
Daan
5b4c4ac959
Merge pull request #117 from devnexen/darwin_anon_monitor
...
darwin, tracking anonymous page lifetime.
2019-08-10 09:58:35 -07:00
daan
ccc78302bf
fix double assignment ( #125 , by @ebudai)
2019-08-10 09:53:00 -07:00
daan
3d9d70bce0
add new redirection module; fix environment reading at preloading
2019-08-10 09:39:36 -07:00
daan
7b16aa9302
remove accidental commit of different size bins experiment
2019-08-09 11:22:38 -07:00
daan
442bad9190
add good-fit for allowing larger blocks in smaller segments
2019-08-09 11:18:38 -07:00
daan
5e56b40fe6
improve page flags handling
2019-08-08 17:18:49 -07:00
daan
de57686dac
Merge branch 'dev-win' of https://github.com/microsoft/mimalloc into dev-win
2019-08-08 15:23:27 -07:00
daan
6596e970a5
move in_full and has_aligned into page threadid for a single test in mi_free
2019-08-08 15:23:18 -07:00
daan
55778d2fe4
improved stats
2019-08-08 11:36:13 -07:00
daan
56778fe7d2
split normal and secure extend in a separate routine
2019-08-06 18:57:53 -07:00
Jakub Szymanski
0e8241c140
change from MIMALLOC_OVERRIDE to MIMALLOC_DISABLE_OVERRIDE
2019-07-30 10:36:00 -07:00
David Carlier
82ab9b4bd6
tracking anonymous page lifetime on macOS.
...
this platform allows to tag them per application up to 255.
2019-07-27 23:05:05 +01:00
daan
bbf97fa02d
merge from 'dev-exp'; add support for MIMALLOC_OVERRIDE environment variable
2019-07-26 12:06:15 -07:00
daan
613d0c1993
merge 095a87b
2019-07-23 18:00:07 -07:00
daan
095a87be2e
add recursion guard for overriding in windows
2019-07-23 17:57:27 -07:00
daan
3d6feead60
add heap region check to cfree
2019-07-23 15:54:47 -07:00
daan
dd79b662f8
restore to old-style windows override; fix debug mode
2019-07-23 15:51:08 -07:00
daan
fa5dc17621
fix merge conflicts
2019-07-23 15:09:54 -07:00
daan
a022049705
remove old comment
2019-07-23 15:00:13 -07:00
daan
189ad0f81d
small optimizations, use bitwise aligne
2019-07-23 15:00:13 -07:00
daan
66b8c37ab3
ensure C++ compilation on windows
2019-07-23 15:00:13 -07:00
daan
0b4d74a566
merge
2019-07-23 15:00:13 -07:00
daan
b611c7fb34
use atomic ops to guard large page tries on windows
2019-07-23 15:00:13 -07:00
daan
7b7c36c8c7
use hinted address to mmap to reduce mmap calls
2019-07-23 14:59:57 -07:00
Jim Huang
b5e26bedb1
Enforce strict include-what-you-use policy
...
The include-what-you-use (IWYU) policy is beneficial to faster
compilation and fewer recompilations. Many build tools, such as GNU make,
provide a mechanism for automatically figuring out what .h files a .cc
file depends on. These mechanisms typically look at #include lines. When
unnecessary #includes are listed, the build system is more likely to
recompile in cases where it is not necessary.
With the enforcement, header file <include/mimalloc.h> no longer
includes <stdlib.h>.
Reference:
https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYU.md
2019-07-23 14:59:57 -07:00
daan
65037d051b
improved debug warning for freeing invalid pointers
2019-07-23 14:59:57 -07:00
daan
2eb607d03f
re-add missing thread_init needed when running in debug mode
2019-07-23 14:59:57 -07:00
Jim Huang
28b874129a
Avoid using strlen function in loop
2019-07-23 14:59:57 -07:00
Jim Huang
3e9c953eea
Add branch prediction hint for mi_option_get
...
mi_option_get is called frequently in stress tests, and the patch adds
extra hint to the compiler to emit instructions that will cause branch
prediction to favour the "likely" side of a jump instruction.
2019-07-23 14:59:49 -07:00
caixiangyue
7266e7006a
fix typo
2019-07-23 14:59:49 -07:00
daan
46b11fa0a4
add comment about large pages
2019-07-23 14:59:49 -07:00
daan
6ca8b3fd89
remove old comment
2019-07-22 21:25:57 -07:00
daan
f0530b6a83
small optimizations, use bitwise aligne
2019-07-22 20:51:12 -07:00
daan
7c26ce9280
ensure C++ compilation on windows
2019-07-22 10:27:14 -07:00
daan
80e09ef44e
merge from dev
2019-07-22 10:16:59 -07:00
daan
e90938fb4b
merge
2019-07-22 10:10:45 -07:00
daan
7555260759
Merge branch 'jserv-include-what-you-use' into dev
2019-07-22 10:07:03 -07:00
daan
959d9289b1
merge and improve to upper case in options
2019-07-22 10:06:45 -07:00
Daan
80400c9946
Merge pull request #111 from jserv/avoid-strlen-in-loop
...
Avoid using strlen function in loop
2019-07-22 09:51:02 -07:00
Daan
cf8f46c263
Merge pull request #110 from jserv/options-branch-predict-hints
...
Add branch prediction hint for mi_option_get
2019-07-22 09:49:44 -07:00
daan
272a8e03e4
use atomic ops to guard large page tries on windows
2019-07-21 17:13:36 -07:00
daan
069f52523f
use hinted address to mmap to reduce mmap calls
2019-07-21 17:08:09 -07:00
Jim Huang
1e27cef873
Enforce strict include-what-you-use policy
...
The include-what-you-use (IWYU) policy is beneficial to faster
compilation and fewer recompilations. Many build tools, such as GNU make,
provide a mechanism for automatically figuring out what .h files a .cc
file depends on. These mechanisms typically look at #include lines. When
unnecessary #includes are listed, the build system is more likely to
recompile in cases where it is not necessary.
With the enforcement, header file <include/mimalloc.h> no longer
includes <stdlib.h>.
Reference:
https://github.com/include-what-you-use/include-what-you-use/blob/master/docs/WhyIWYU.md
2019-07-22 04:54:14 +08:00
daan
df33efb19a
improved debug warning for freeing invalid pointers
2019-07-21 13:09:34 -07:00
daan
40cb631683
re-add missing thread_init needed when running in debug mode
2019-07-21 13:03:51 -07:00
Jim Huang
c382c72cf2
Avoid using strlen function in loop
2019-07-21 22:30:42 +08:00
Jim Huang
1ffa48cc61
Add branch prediction hint for mi_option_get
...
mi_option_get is called frequently in stress tests, and the patch adds
extra hint to the compiler to emit instructions that will cause branch
prediction to favour the "likely" side of a jump instruction.
2019-07-21 21:42:00 +08:00
daan
f0de0b6f68
remove old windows override code
2019-07-19 09:33:14 -07:00
daan
2ac2c6a1c3
fix format of error messages
2019-07-19 09:27:32 -07:00
daan
fcd0253ac9
limit number of error messages to 10
2019-07-19 09:27:14 -07:00
daan
dd5fa88c45
check better for valid pointers in free in debug mode
2019-07-19 09:26:51 -07:00
caixiangyue
a215049b4a
fix typo
2019-07-19 16:23:14 +08:00
