我有大量并行进程运行.似乎仅仅创建了并行scala Futures会造成内存泄漏.
示例代码如下.根据您的机器设置VM标志,以便在您有多个内核时,它不会在开始时直接转储," - Xmx100m -XX:+ HeapDumpOnOutOfMemoryError".
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits.global
import scala.concurrent.Await
import scala.concurrent.duration._
object Bug extends App {
println("Memory leak")
while(true){
Future {
val data = new Array[Byte](1000000*1) // 1 MB
println(".")
}
}
}
几分钟后,此代码将显着减慢.堆转储分析显示数组scala.concurrent.forkjoin.ForkJoinTask []似乎溢出.
以下链接似乎相关,但并未真正提供任何解决方案:https://issues.scala-lang.org/browse/SI-7336
这里发生的事情是你分配内存比GC更快.
在while循环的每次执行中,您将一个任务放到线程池中以分配1MB,您可以非常快速地分配内存.
过了一会儿,GC正在放缓.
解？分配内存较慢.
顺便说一句,即使你没有声明数组,它也会以类似的方式运行.当你正在运行一个Future你正在向task线程池的队列提交时,所以内存将以非常快的速度增加.
要自己查看,可以运行jstat(http://www.cubrid.org/blog/dev-platform/how-to-monitor-java-garbage-collection/)
jstat -gc $pid 1000
这是我笔记本电脑的样本,看看GCT列(GC操作的总累计时间)是如何增加的
? ~ jstat -gc 24504 1000 S0C S1C S0U S1U EC EU OC OU MC MU CCSC CCSU YGC YGCT FGC FGCT GCT 35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458 35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458 35328.0 35840.0 22962.9 0.0 275968.0 0.0 395264.0 383080.0 11904.0 11387.3 1664.0 1545.4 30 1.333 3 2.125 3.458 36352.0 36864.0 8148.0 0.0 275456.0 0.0 562688.0 533891.1 11904.0 11407.6 1664.0 1548.2 36 1.772 4 4.751 6.524 36352.0 36864.0 8148.0 0.0 275456.0 0.0 562688.0 533891.1 11904.0 11407.6 1664.0 1548.2 36 1.772 4 4.751 6.524 36352.0 36864.0 0.0 0.0 275456.0 143402.5 699392.0 486444.1 11904.0 11407.6 1664.0 1548.2 37 1.772 4 6.867 8.640 32768.0 33792.0 22498.3 11986.9 278528.0 278251.8 699392.0 632212.1 11904.0 11437.5 1664.0 1549.8 47 2.548 4 6.867 9.415 29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635 29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635 29696.0 31232.0 20498.9 0.0 284160.0 0.0 699392.0 678668.1 11904.0 11448.8 1664.0 1551.3 50 2.768 5 6.867 9.635 29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220 29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220 29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220 29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220 29696.0 31232.0 0.0 0.0 284160.0 283932.1 699392.0 691190.3 11904.0 11448.8 1664.0 1551.3 50 2.768 6 10.452 13.220 29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837 29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837 29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837 29696.0 31232.0 0.0 0.0 284160.0 283918.2 699392.0 698585.9 11904.0 11451.8 1664.0 1551.3 50 2.768 7 15.069 17.837 29696.0 31232.0 0.0 0.0 284160.0 16821.1 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 7 19.562 22.330 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699140.7 11904.0 11452.4 1664.0 1551.3 50 2.768 8 19.562 22.330 29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880 29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880 29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880 29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880 29696.0 31232.0 0.0 0.0 284160.0 284029.3 699392.0 699116.2 11904.0 11453.7 1664.0 1551.3 50 2.768 9 24.112 26.880 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699092.9 11904.0 11455.9 1664.0 1551.3 50 2.768 10 28.728 31.496 29696.0 31232.0 0.0 0.0 284160.0 176242.5 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699074.7 11904.0 11455.9 1664.0 1551.3 50 2.768 11 33.550 36.318 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565 29696.0 31232.0 0.0 0.0 284160.0 284160.0 699392.0 699313.9 11904.0 11458.4 1664.0 1551.3 50 2.768 12 38.797 41.565
京公网安备 11010802040832号 | 京ICP备19059560号-6 