Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make json parsing more efficient on the JSON unfolding feature #34798

Closed
ericcj opened this issue Oct 18, 2023 · 6 comments · Fixed by #42638
Closed

Make json parsing more efficient on the JSON unfolding feature #34798

ericcj opened this issue Oct 18, 2023 · 6 comments · Fixed by #42638
Assignees
@piranha
Labels
Administration/Metadata & Sync .Escalation .Performance Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness .Team/BackendComponents also known as BEC Type:Bug Product defects
Milestone
0.49.11

Comments

@ericcj
Copy link
Contributor

ericcj commented Oct 18, 2023

Describe the bug

Same problem as #34754 (comment) but I upgraded the instance type to have 4GB and set JAVA_OPTS=-Xmx2g and now it just takes longer to crash, pegging the CPU to 100% for 10m before crashing. Everything works fine without "Allow unfolding of JSON columns" and i can't figure out which table might be the problem since removing permissions for 5 that it seemed to have selected from last didn't fix the problem.

To Reproduce

Click "Sync database schema now"
Watch logs to see OutOfMemoryError

Expected behavior

No response

Logs

[15abff80-c314-4109-970a-8e7d254692e6] 2023-10-18T10:51:12-04:00 INFO metabase.sync.util STARTING: step 'sync-fields' for postgres Database 2 'Application'
...
2023-10-18 14:58:15,659 INFO middleware.exceptions :: Request canceled before finishing.
Aborting due to java.lang.OutOfMemoryError: Java heap space
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (debug.cpp:339), pid=1, tid=84
#  fatal error: OutOfMemory encountered: Java heap space
#
# JRE version: OpenJDK Runtime Environment Temurin-11.0.20.1+1 (11.0.20.1+1) (build 11.0.20.1+1)
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.20.1+1 (11.0.20.1+1, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Core dump will be written. Default location: //core.1
#

---------------  S U M M A R Y ------------

Command Line: -Xmx2g -XX:+IgnoreUnrecognizedVMOptions -Dfile.encoding=UTF-8 -Dlogfile.path=target/log -XX:+CrashOnOutOfMemoryError /app/metabase.jar

Host: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz, 2 cores, 3G, Alpine Linux v3.18
Time: Wed Oct 18 14:58:45 2023 UTC elapsed time: 893.616185 seconds (0d 0h 14m 53s)

---------------  T H R E A D  ---------------

Current thread (0x00007f6862664800):  JavaThread "clojure-agent-send-off-pool-7" [_thread_in_vm, id=84, stack(0x00007f68609fb000,0x00007f6860afbaa8)]

Stack: [0x00007f68609fb000,0x00007f6860afbaa8],  sp=0x00007f6860afa380,  free space=1020k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xea9476]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1c6
V  [libjvm.so+0x6b5dd4]  report_fatal(VMErrorType, char const*, int, char const*, ...)+0x104
V  [libjvm.so+0x6b6035]  report_java_out_of_memory(char const*)+0xb5
V  [libjvm.so+0xb7af8e]  MemAllocator::allocate() const+0x12e
V  [libjvm.so+0x62b544]  CollectedHeap::array_allocate(Klass*, int, int, bool, Thread*)+0x44
V  [libjvm.so+0x87d535]  InstanceKlass::allocate_objArray(int, int, Thread*)+0x115
V  [libjvm.so+0x56e489]  Runtime1::new_object_array(JavaThread*, Klass*, int)+0x89
v  ~RuntimeStub::new_object_array Runtime1 stub
J 12286 c1 com.fasterxml.jackson.core.util.BufferRecyclers.getBufferRecycler()Lcom/fasterxml/jackson/core/util/BufferRecycler; (73 bytes) @ 0x00007f687e3fe46b [0x00007f687e3fd4e0+0x0000000000000f8b]
J 11760 c1 com.fasterxml.jackson.core.JsonFactory._createContext(Lcom/fasterxml/jackson/core/io/ContentReference;Z)Lcom/fasterxml/jackson/core/io/IOContext; (22 bytes) @ 0x00007f687e235d1c [0x00007f687e2359c0+0x000000000000035c]
j  com.fasterxml.jackson.core.JsonFactory.createGenerator(Ljava/io/Writer;)Lcom/fasterxml/jackson/core/JsonGenerator;+7
j  cheshire.core$generate_stream.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+38
j  cheshire.core$generate_stream.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j  metabase.server.middleware.json$streamed_json_response$fn__113192.invoke(Ljava/lang/Object;)Ljava/lang/Object;+50
j  ring.util.io$piped_input_stream$fn__683.invoke()Ljava/lang/Object;+16
J 14761 c1 clojure.core$binding_conveyor_fn$fn__5823.invoke()Ljava/lang/Object; (22 bytes) @ 0x00007f687d2cbd7c [0x00007f687d2cb6c0+0x00000000000006bc]
J 14714 c2 clojure.lang.AFn.call()Ljava/lang/Object; (5 bytes) @ 0x00007f6883efcfdc [0x00007f6883efcfa0+0x000000000000003c]
J 12004 c1 java.util.concurrent.FutureTask.run()V java.base@11.0.20.1 (123 bytes) @ 0x00007f687e30151c [0x00007f687e300ea0+0x000000000000067c]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@11.0.20.1
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@11.0.20.1
j  java.lang.Thread.run()V+11 java.base@11.0.20.1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x8a4ab3]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x373
V  [libjvm.so+0x8a2fc6]  JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Thread*)+0x1a6
V  [libjvm.so+0x95508f]  thread_entry(JavaThread*, Thread*)+0x9f
V  [libjvm.so+0xe404c3]  JavaThread::thread_main_inner()+0x133
V  [libjvm.so+0xe3c8dc]  Thread::call_run()+0x13c
V  [libjvm.so+0xc26de6]  thread_native_entry(Thread*)+0x136

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  ~RuntimeStub::new_object_array Runtime1 stub
J 12286 c1 com.fasterxml.jackson.core.util.BufferRecyclers.getBufferRecycler()Lcom/fasterxml/jackson/core/util/BufferRecycler; (73 bytes) @ 0x00007f687e3fe46b [0x00007f687e3fd4e0+0x0000000000000f8b]
J 11760 c1 com.fasterxml.jackson.core.JsonFactory._createContext(Lcom/fasterxml/jackson/core/io/ContentReference;Z)Lcom/fasterxml/jackson/core/io/IOContext; (22 bytes) @ 0x00007f687e235d1c [0x00007f687e2359c0+0x000000000000035c]
j  com.fasterxml.jackson.core.JsonFactory.createGenerator(Ljava/io/Writer;)Lcom/fasterxml/jackson/core/JsonGenerator;+7
j  cheshire.core$generate_stream.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+38
j  cheshire.core$generate_stream.invoke(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+9
j  metabase.server.middleware.json$streamed_json_response$fn__113192.invoke(Ljava/lang/Object;)Ljava/lang/Object;+50
j  ring.util.io$piped_input_stream$fn__683.invoke()Ljava/lang/Object;+16
J 14761 c1 clojure.core$binding_conveyor_fn$fn__5823.invoke()Ljava/lang/Object; (22 bytes) @ 0x00007f687d2cbd7c [0x00007f687d2cb6c0+0x00000000000006bc]
J 14714 c2 clojure.lang.AFn.call()Ljava/lang/Object; (5 bytes) @ 0x00007f6883efcfdc [0x00007f6883efcfa0+0x000000000000003c]
J 12004 c1 java.util.concurrent.FutureTask.run()V java.base@11.0.20.1 (123 bytes) @ 0x00007f687e30151c [0x00007f687e300ea0+0x000000000000067c]
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+92 java.base@11.0.20.1
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 java.base@11.0.20.1
j  java.lang.Thread.run()V+11 java.base@11.0.20.1
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00007f6870dfb190, length=117, elements={
0x00007f6890cad800, 0x00007f6873fcc800, 0x00007f6873db2800, 0x00007f6873db3800,
...
0x00007f6860ed4800
}

Java Threads: ( => current thread )
  0x00007f6890cad800 JavaThread "main" [_thread_blocked, id=18, stack(0x00007f6890d46000,0x00007f6890e46ad8)]
  0x00007f6873fcc800 JavaThread "Reference Handler" daemon [_thread_blocked, id=25, stack(0x00007f6873dbe000,0x00007f6873ebeac0)]
  0x00007f6873db2800 JavaThread "Finalizer" daemon [_thread_blocked, id=26, stack(0x00007f6873cb1000,0x00007f6873db1ac0)]
  0x00007f6873db3800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=27, stack(0x00007f6873b86000,0x00007f6873c86ac0)]
  0x00007f6873c9f800 JavaThread "Service Thread" daemon [_thread_blocked, id=28, stack(0x00007f6873a85000,0x00007f6873b85ac0)]
  0x00007f6873ca0800 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=29, stack(0x00007f6873984000,0x00007f6873a84ac0)]
  0x00007f6873c9a800 JavaThread "C1 CompilerThread0" daemon [_thread_blocked, id=30, stack(0x00007f6873883000,0x00007f6873983ac0)]
  0x00007f6873c9b800 JavaThread "Sweeper thread" daemon [_thread_blocked, id=31, stack(0x00007f6873782000,0x00007f6873882ac0)]
  0x00007f687353a800 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=33, stack(0x00007f6873439000,0x00007f6873539ac0)]
  0x00007f686fdd2800 JavaThread "sshd-SshClient[7116940a]-timer-thread-1" daemon [_thread_blocked, id=37, stack(0x00007f686b80a000,0x00007f686b90aaa8)]
  0x00007f686fdd0800 JavaThread "Thread-0" daemon [_thread_in_native, id=38, stack(0x00007f686b60f000,0x00007f686b70faa8)]
  0x00007f686d943800 JavaThread "async-dispatch-1" daemon [_thread_blocked, id=39, stack(0x00007f686ca33000,0x00007f686cb33aa8)]
  0x00007f686ff93800 JavaThread "qtp436560927-15" [_thread_blocked, id=40, stack(0x00007f6867775000,0x00007f6867875aa8)]
  0x00007f686ff95800 JavaThread "qtp436560927-16-acceptor-0@131dd6bd-ServerConnector@1dda3aae{HTTP/1.1, (http/1.1)}{0.0.0.0:3000}" [_thread_blocked, id=41, stack(0x00007f6867674000,0x00007f6867774aa8)]
  0x00007f686ff96800 JavaThread "qtp436560927-17" [_thread_blocked, id=42, stack(0x00007f6867573000,0x00007f6867673aa8)]
  0x00007f686ff97800 JavaThread "qtp436560927-18" [_thread_blocked, id=43, stack(0x00007f6867472000,0x00007f6867572aa8)]
  0x00007f68682d6800 JavaThread "qtp436560927-19" [_thread_blocked, id=44, stack(0x00007f6867371000,0x00007f6867471aa8)]
  0x00007f68682d7800 JavaThread "qtp436560927-20" [_thread_blocked, id=45, stack(0x00007f6867270000,0x00007f6867370aa8)]
  0x00007f68682d8800 JavaThread "qtp436560927-21" [_thread_blocked, id=46, stack(0x00007f686712c000,0x00007f686722caa8)]
  0x00007f68682d9800 JavaThread "qtp436560927-22" [_thread_blocked, id=47, stack(0x00007f686702b000,0x00007f686712baa8)]
  0x00007f68682db800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-AdminTaskTimer" daemon [_thread_blocked, id=48, stack(0x00007f6866e2c000,0x00007f6866f2caa8)]
  0x00007f68682dc000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-HelperThread-#0" daemon [_thread_blocked, id=49, stack(0x00007f6865bc2000,0x00007f6865cc2aa8)]
  0x00007f6867eaf800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-HelperThread-#2" daemon [_thread_blocked, id=51, stack(0x00007f68659c0000,0x00007f6865ac0aa8)]
  0x00007f6867eb2800 JavaThread "Scheduler-1946879397-1" [_thread_blocked, id=52, stack(0x00007f6864d24000,0x00007f6864e24aa8)]
  0x00007f6867eb3800 JavaThread "clojure-agent-send-off-pool-0" [_thread_blocked, id=53, stack(0x00007f6864683000,0x00007f6864783aa8)]
  0x00007f6867eb4800 JavaThread "clojure-agent-send-off-pool-1" [_thread_blocked, id=54, stack(0x00007f6864b89000,0x00007f6864c89aa8)]
  0x00007f68652c2800 JavaThread "async-dispatch-2" daemon [_thread_blocked, id=55, stack(0x00007f6864382000,0x00007f6864482aa8)]
  0x00007f68652c3800 JavaThread "async-dispatch-3" daemon [_thread_blocked, id=56, stack(0x00007f6864281000,0x00007f6864381aa8)]
  0x00007f68652c4800 JavaThread "async-dispatch-4" daemon [_thread_blocked, id=57, stack(0x00007f6864180000,0x00007f6864280aa8)]
  0x00007f68652c5800 JavaThread "async-dispatch-5" daemon [_thread_blocked, id=58, stack(0x00007f686407f000,0x00007f686417faa8)]
  0x00007f68652c6800 JavaThread "async-dispatch-6" daemon [_thread_blocked, id=59, stack(0x00007f6863f7e000,0x00007f686407eaa8)]
  0x00007f68652c7800 JavaThread "async-dispatch-7" daemon [_thread_blocked, id=60, stack(0x00007f6863e7d000,0x00007f6863f7daa8)]
  0x00007f68652c8800 JavaThread "async-dispatch-8" daemon [_thread_blocked, id=61, stack(0x00007f6863d7c000,0x00007f6863e7caa8)]
  0x00007f68652c9000 JavaThread "MetabaseScheduler_Worker-1" [_thread_blocked, id=62, stack(0x00007f6863c7b000,0x00007f6863d7baa8)]
  0x00007f68650f9800 JavaThread "MetabaseScheduler_Worker-2" [_thread_blocked, id=63, stack(0x00007f6863b7a000,0x00007f6863c7aaa8)]
  0x00007f68650fa800 JavaThread "MetabaseScheduler_Worker-3" [_thread_blocked, id=64, stack(0x00007f6863a79000,0x00007f6863b79aa8)]
  0x00007f68650fb800 JavaThread "MetabaseScheduler_Worker-4" [_thread_blocked, id=65, stack(0x00007f6863978000,0x00007f6863a78aa8)]
  0x00007f68650fc800 JavaThread "MetabaseScheduler_Worker-5" [_thread_blocked, id=66, stack(0x00007f6863877000,0x00007f6863977aa8)]
  0x00007f68650fd800 JavaThread "MetabaseScheduler_Worker-6" [_thread_blocked, id=67, stack(0x00007f6863776000,0x00007f6863876aa8)]
  0x00007f68650fe800 JavaThread "MetabaseScheduler_Worker-7" [_thread_blocked, id=68, stack(0x00007f6863675000,0x00007f6863775aa8)]
  0x00007f68650ff800 JavaThread "MetabaseScheduler_Worker-8" [_thread_blocked, id=69, stack(0x00007f6863574000,0x00007f6863674aa8)]
  0x00007f6865100800 JavaThread "MetabaseScheduler_Worker-9" [_thread_blocked, id=70, stack(0x00007f6863473000,0x00007f6863573aa8)]
  0x00007f6865101800 JavaThread "MetabaseScheduler_Worker-10" [_thread_blocked, id=71, stack(0x00007f6863372000,0x00007f6863472aa8)]
  0x00007f6865102000 JavaThread "MetabaseScheduler_QuartzSchedulerThread" [_thread_blocked, id=72, stack(0x00007f6863271000,0x00007f6863371aa8)]
  0x00007f6865104000 JavaThread "QuartzScheduler_MetabaseScheduler-c796feeeeacf1697640267643_ClusterManager" [_thread_blocked, id=73, stack(0x00007f6862960000,0x00007f6862a60aa8)]
  0x00007f6865105800 JavaThread "QuartzScheduler_MetabaseScheduler-c796feeeeacf1697640267643_MisfireHandler" [_thread_blocked, id=74, stack(0x00007f686285f000,0x00007f686295faa8)]
  0x00007f686311a000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|6639926b, dataSourceName->db-2-postgres-steven_prod]-AdminTaskTimer" daemon [_thread_blocked, id=75, stack(0x00007f6862665000,0x00007f6862765aa8)]
  0x00007f686311b800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|6639926b, dataSourceName->db-2-postgres-steven_prod]-HelperThread-#0" daemon [_thread_blocked, id=76, stack(0x00007f6861f17000,0x00007f6862017aa8)]
  0x00007f686311c800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|6639926b, dataSourceName->db-2-postgres-steven_prod]-HelperThread-#1" daemon [_thread_blocked, id=77, stack(0x00007f6861e16000,0x00007f6861f16aa8)]
  0x00007f686311d800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|6639926b, dataSourceName->db-2-postgres-steven_prod]-HelperThread-#2" daemon [_thread_blocked, id=78, stack(0x00007f6861d15000,0x00007f6861e15aa8)]
  0x00007f6863122800 JavaThread "clojure-agent-send-off-pool-2" [_thread_blocked, id=79, stack(0x00007f6861a14000,0x00007f6861b14aa8)]
  0x00007f686265d800 JavaThread "clojure-agent-send-off-pool-3" [_thread_blocked, id=80, stack(0x00007f68613b7000,0x00007f68614b7aa8)]
  0x00007f686265e800 JavaThread "clojure-agent-send-off-pool-5" [_thread_blocked, id=81, stack(0x00007f68612b6000,0x00007f68613b6aa8)]
  0x00007f686265f800 JavaThread "clojure-agent-send-off-pool-4" [_thread_blocked, id=82, stack(0x00007f68611b5000,0x00007f68612b5aa8)]
  0x00007f6862660800 JavaThread "clojure-agent-send-off-pool-6" [_thread_blocked, id=83, stack(0x00007f6860fef000,0x00007f68610efaa8)]
=>0x00007f6862664800 JavaThread "clojure-agent-send-off-pool-7" [_thread_in_vm, id=84, stack(0x00007f68609fb000,0x00007f6860afbaa8)]
  0x00007f6860b26800 JavaThread "clojure-agent-send-off-pool-8" [_thread_blocked, id=85, stack(0x00007f68608fa000,0x00007f68609faaa8)]
  0x00007f6860b27800 JavaThread "qtp436560927-62" [_thread_blocked, id=86, stack(0x00007f68606cb000,0x00007f68607cbaa8)]
  0x00007f6860b28800 JavaThread "qtp436560927-63" [_thread_blocked, id=87, stack(0x00007f686022a000,0x00007f686032aaa8)]
  0x00007f6860b29800 JavaThread "qtp436560927-64" [_thread_blocked, id=88, stack(0x00007f6860129000,0x00007f6860229aa8)]
  0x00007f6860b2a800 JavaThread "qtp436560927-65" [_thread_blocked, id=89, stack(0x00007f6860028000,0x00007f6860128aa8)]
  0x00007f6860b2b800 JavaThread "qtp436560927-66" [_thread_blocked, id=90, stack(0x00007f6860cb6000,0x00007f6860db6aa8)]
  0x00007f6860b2c800 JavaThread "qtp436560927-67" [_thread_blocked, id=91, stack(0x00007f6860bb5000,0x00007f6860cb5aa8)]
  0x00007f6860b2d800 JavaThread "qtp436560927-68" [_thread_blocked, id=92, stack(0x00007f68607f9000,0x00007f68608f9aa8)]
  0x00007f6860b2e800 JavaThread "qtp436560927-69" [_thread_blocked, id=93, stack(0x00007f68605ca000,0x00007f68606caaa8)]
  0x00007f6860b2f800 JavaThread "qtp436560927-70" [_thread_blocked, id=94, stack(0x00007f68604c9000,0x00007f68605c9aa8)]
  0x00007f6860b30800 JavaThread "qtp436560927-71" [_thread_blocked, id=95, stack(0x00007f68603c8000,0x00007f68604c8aa8)]
  0x00007f6860b31800 JavaThread "qtp436560927-72" [_thread_blocked, id=96, stack(0x00007f685fc27000,0x00007f685fd27aa8)]
  0x00007f6860b32000 JavaThread "qtp436560927-73" [_thread_blocked, id=97, stack(0x00007f685ff27000,0x00007f6860027aa8)]
  0x00007f6860b33800 JavaThread "qtp436560927-74" [_thread_blocked, id=98, stack(0x00007f685fb26000,0x00007f685fc26aa8)]
  0x00007f6860b34000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-HelperThread-#0" daemon [_thread_blocked, id=99, stack(0x00007f685fa25000,0x00007f685fb25aa8)]
  0x00007f6860b35000 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-HelperThread-#1" daemon [_thread_blocked, id=100, stack(0x00007f685f924000,0x00007f685fa24aa8)]
  0x00007f6861118800 JavaThread "C3P0PooledConnectionPoolManager[identityToken->1bqq1heazinogdc4lnuvx|7b9b32f3, dataSourceName->metabase-postgres-app-db]-HelperThread-#2" daemon [_thread_blocked, id=101, stack(0x00007f685f823000,0x00007f685f923aa8)]
  0x00007f6861119800 JavaThread "Dispatch-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=102, stack(0x00007f685f722000,0x00007f685f822aa8)]
  0x00007f686111a800 JavaThread "qtp436560927-79" [_thread_blocked, id=103, stack(0x00007f685fe26000,0x00007f685ff26aa8)]
  0x00007f686111e800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=107, stack(0x00007f685f021000,0x00007f685f121aa8)]
  0x00007f686111f800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=108, stack(0x00007f685ef20000,0x00007f685f020aa8)]
  0x00007f6861121800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=110, stack(0x00007f685e81e000,0x00007f685e91eaa8)]
  0x00007f6861122800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=111, stack(0x00007f685e71d000,0x00007f685e81daa8)]
  0x00007f6861123800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=112, stack(0x00007f685e61c000,0x00007f685e71caa8)]
  0x00007f6861124800 JavaThread "Task-Thread-for-com.mchange.v2.async.ThreadPerTaskAsynchronousRunner@3bc04e94" daemon [_thread_blocked, id=113, stack(0x00007f685e51b000,0x00007f685e61baa8)]
  0x00007f686111c800 JavaThread "qtp436560927-91" [_thread_blocked, id=115, stack(0x00007f685f520000,0x00007f685f620aa8)]
  0x00007f6861120800 JavaThread "qtp436560927-92" [_thread_blocked, id=116, stack(0x00007f685f31e000,0x00007f685f41eaa8)]
  0x00007f6861125000 JavaThread "qtp436560927-93" [_thread_blocked, id=117, stack(0x00007f685f21d000,0x00007f685f31daa8)]
  0x00007f6861126000 JavaThread "qtp436560927-94" [_thread_blocked, id=118, stack(0x00007f685ee1f000,0x00007f685ef1faa8)]
  0x00007f6861127000 JavaThread "qtp436560927-95" [_thread_blocked, id=119, stack(0x00007f685ed1e000,0x00007f685ee1eaa8)]
  0x00007f6861128000 JavaThread "qtp436560927-96" [_thread_blocked, id=120, stack(0x00007f685ec1d000,0x00007f685ed1daa8)]
  0x00007f6861129000 JavaThread "qtp436560927-97" [_thread_blocked, id=121, stack(0x00007f685e91f000,0x00007f685ea1faa8)]
  0x00007f686112a000 JavaThread "qtp436560927-98" [_thread_blocked, id=122, stack(0x00007f685e41a000,0x00007f685e51aaa8)]
  0x00007f686112b000 JavaThread "qtp436560927-99" [_thread_blocked, id=123, stack(0x00007f685e319000,0x00007f685e419aa8)]
  0x00007f686112c000 JavaThread "qtp436560927-100" [_thread_blocked, id=124, stack(0x00007f685eb1c000,0x00007f685ec1caa8)]
  0x00007f686112d000 JavaThread "qtp436560927-101" [_thread_blocked, id=125, stack(0x00007f685e218000,0x00007f685e318aa8)]
  0x00007f686112e000 JavaThread "qtp436560927-102" [_thread_blocked, id=126, stack(0x00007f685e117000,0x00007f685e217aa8)]
  0x00007f686112f000 JavaThread "qtp436560927-103" [_thread_blocked, id=127, stack(0x00007f685e016000,0x00007f685e116aa8)]
  0x00007f6861130000 JavaThread "qtp436560927-104" [_thread_blocked, id=128, stack(0x00007f685df15000,0x00007f685e015aa8)]
  0x00007f6861131000 JavaThread "qtp436560927-105" [_thread_blocked, id=129, stack(0x00007f685d417000,0x00007f685d517aa8)]
  0x00007f6861132000 JavaThread "qtp436560927-106" [_thread_blocked, id=130, stack(0x00007f685d316000,0x00007f685d416aa8)]
  0x00007f6861133000 JavaThread "qtp436560927-107" [_thread_blocked, id=131, stack(0x00007f685d016000,0x00007f685d116aa8)]
  0x00007f6861134000 JavaThread "qtp436560927-108" [_thread_blocked, id=132, stack(0x00007f685cf15000,0x00007f685d015aa8)]
  0x00007f6861135000 JavaThread "qtp436560927-109" [_thread_blocked, id=133, stack(0x00007f685ce14000,0x00007f685cf14aa8)]
  0x00007f6861137000 JavaThread "qtp436560927-111" [_thread_blocked, id=135, stack(0x00007f685cc12000,0x00007f685cd12aa8)]
  0x00007f6860ec8800 JavaThread "qtp436560927-112" [_thread_blocked, id=136, stack(0x00007f685cb11000,0x00007f685cc11aa8)]
  0x00007f6860ec9800 JavaThread "clojure-agent-send-off-pool-9" [_thread_blocked, id=137, stack(0x00007f685de14000,0x00007f685df14aa8)]
  0x00007f6860eca800 JavaThread "clojure-agent-send-off-pool-10" [_thread_blocked, id=138, stack(0x00007f685dd13000,0x00007f685de13aa8)]
  0x00007f6860ecc800 JavaThread "clojure-agent-send-off-pool-11" [_thread_blocked, id=139, stack(0x00007f685dc12000,0x00007f685dd12aa8)]
  0x00007f6860ecd800 JavaThread "clojure-agent-send-off-pool-12" [_thread_blocked, id=140, stack(0x00007f685db11000,0x00007f685dc11aa8)]
  0x00007f686111b800 JavaThread "qtp436560927-117" [_thread_blocked, id=141, stack(0x00007f685f621000,0x00007f685f721aa8)]
  0x00007f686111d800 JavaThread "qtp436560927-118" [_thread_blocked, id=142, stack(0x00007f685f41f000,0x00007f685f51faa8)]
  0x00007f6861136000 JavaThread "qtp436560927-119" [_thread_blocked, id=143, stack(0x00007f685da10000,0x00007f685db10aa8)]
  0x00007f68682dd800 JavaThread "qtp436560927-120" [_thread_blocked, id=144, stack(0x00007f685d90f000,0x00007f685da0faa8)]
  0x00007f6860ece800 JavaThread "qtp436560927-122" [_thread_blocked, id=145, stack(0x00007f685d80e000,0x00007f685d90eaa8)]
  0x00007f6860ecf800 JavaThread "qtp436560927-123" [_thread_blocked, id=146, stack(0x00007f685d70d000,0x00007f685d80daa8)]
  0x00007f6860ed0800 JavaThread "qtp436560927-121" [_thread_blocked, id=147, stack(0x00007f685d60c000,0x00007f685d70caa8)]
  0x00007f6860ed1000 JavaThread "clojure-agent-send-off-pool-13" [_thread_blocked, id=148, stack(0x00007f685cd13000,0x00007f685ce13aa8)]
  0x00007f6860ed2800 JavaThread "clojure-agent-send-off-pool-14" [_thread_blocked, id=149, stack(0x00007f685c710000,0x00007f685c810aa8)]
  0x00007f6860ed3000 JavaThread "clojure-agent-send-off-pool-15" [_thread_blocked, id=150, stack(0x00007f685d215000,0x00007f685d315aa8)]
  0x00007f6860ed4800 JavaThread "clojure-agent-send-off-pool-16" [_thread_blocked, id=151, stack(0x00007f685ca10000,0x00007f685cb10aa8)]

Other Threads:
  0x00007f6873fcb800 VMThread "VM Thread" [stack: 0x00007f6873eca000,0x00007f6873fcaac0] [id=24]
  0x00007f6873643800 WatcherThread [stack: 0x00007f687353e000,0x00007f687363eac0] [id=32]
  0x00007f687bf6f800 GCTaskThread "GC Thread#0" [stack: 0x00007f687be6e000,0x00007f687bf6eac0] [id=19]
  0x00007f6873645000 GCTaskThread "GC Thread#1" [stack: 0x00007f6872b7d000,0x00007f6872c7dac0] [id=34]
  0x00007f6876fe5800 ConcurrentGCThread "G1 Main Marker" [stack: 0x00007f6876ee4000,0x00007f6876fe4ac0] [id=20]
  0x00007f6876ee1800 ConcurrentGCThread "G1 Conc#0" [stack: 0x00007f6876de0000,0x00007f6876ee0ac0] [id=21]
  0x00007f6874b89800 ConcurrentGCThread "G1 Refine#0" [stack: 0x00007f6874a86000,0x00007f6874b86ac0] [id=22]
  0x00007f6871ffe800 ConcurrentGCThread "G1 Refine#1" [stack: 0x00007f687118d000,0x00007f687128dac0] [id=35]
  0x00007f6874a83800 ConcurrentGCThread "G1 Young RemSet Sampling" [stack: 0x00007f6874982000,0x00007f6874a82ac0] [id=23]

Threads with active compile tasks:
C2 CompilerThread0   893672 14986   !   4       java.util.jar.Attributes::read (354 bytes)

VM state:synchronizing (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00007f6890cba970] Threads_lock - owner thread: 0x00007f6873fcb800
[0x00007f6890cb72f0] Heap_lock - owner thread: 0x00007f6861118800

Heap address: 0x0000000080000000, size: 2048 MB, Compressed Oops mode: 32-bit
Narrow klass base: 0x0000000000000000, Narrow klass shift: 3
Compressed class space size: 1073741824 Address: 0x0000000100000000

Heap:
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
Heap Regions: E=young(eden), S=young(survivor), O=old, HS=humongous(starts), HC=humongous(continues), CS=collection set, F=free, A=archive, TAMS=top-at-mark-start (previous, next)
|   0|0x0000000080000000, 0x00000000800ffff8, 0x0000000080100000| 99%| O|  |TAMS 0x0000000080000000, 0x0000000080000000| Untracked 
...
|2047|0x00000000fff00000, 0x00000000fffff678, 0x0000000100000000| 99%| O|  |TAMS 0x00000000fff00000, 0x00000000fff00000| Untracked 

Card table byte_map: [0x00007f687b44f000,0x00007f687b84f000] _byte_map_base: 0x00007f687b04f000

Marking Bits (Prev, Next): (CMBitMap*) 0x00007f6877048890, (CMBitMap*) 0x00007f6877048858
 Prev Bits: [0x00007f687704f000, 0x00007f687904f000)
 Next Bits: [0x00007f687904f000, 0x00007f687b04f000)

Polling page: 0x00007f6890cca000

Metaspace:

Usage:
  Non-class:    160.57 MB capacity,   159.43 MB (>99%) used,   925.40 KB ( <1%) free+waste,   238.94 KB ( <1%) overhead. 
      Class:     40.53 MB capacity,    39.76 MB ( 98%) used,   659.98 KB (  2%) free+waste,   128.00 KB ( <1%) overhead. 
       Both:    201.10 MB capacity,   199.20 MB (>99%) used,     1.55 MB ( <1%) free+waste,   366.94 KB ( <1%) overhead. 

Virtual space:
  Non-class space:      164.00 MB reserved,     163.28 MB (>99%) committed 
      Class space:        1.00 GB reserved,      40.90 MB (  4%) committed 
             Both:        1.16 GB reserved,     204.18 MB ( 17%) committed 

Chunk freelists:
   Non-Class:  2.50 MB
       Class:  352.00 KB
        Both:  2.84 MB

MaxMetaspaceSize: unlimited
CompressedClassSpaceSize: 1.00 GB
Initial GC threshold: 20.80 MB
Current GC threshold: 340.29 MB
CDS: off

CodeHeap 'non-profiled nmethods': size=120036Kb used=8713Kb max_used=8713Kb free=111322Kb
 bounds [0x00007f6883c25000, 0x00007f68844b5000, 0x00007f688b15e000]
CodeHeap 'profiled nmethods': size=120032Kb used=34381Kb max_used=35321Kb free=85650Kb
 bounds [0x00007f687c6ed000, 0x00007f687e99d000, 0x00007f6883c25000]
CodeHeap 'non-nmethods': size=5692Kb used=1390Kb max_used=1434Kb free=4301Kb
 bounds [0x00007f687c15e000, 0x00007f687c3ce000, 0x00007f687c6ed000]
 total_blobs=13787 nmethods=13088 adapters=611
 compilation: enabled
              stopped_count=0, restarted_count=0
 full_count=0

Compilation events (20 events):
Event: 883.126 Thread 0x00007f6873c9a800 14993       3       sun.security.util.DisabledAlgorithmConstraints::getNamedCurveFromKey (71 bytes)
Event: 883.127 Thread 0x00007f6873c9a800 nmethod 14993 0x00007f687c75b390 code [0x00007f687c75b6a0, 0x00007f687c75c9d0]
Event: 883.127 Thread 0x00007f6873c9a800 14994       3       sun.nio.cs.StreamEncoder::<init> (56 bytes)
Event: 883.128 Thread 0x00007f6873c9a800 nmethod 14994 0x00007f687cf67910 code [0x00007f687cf67b20, 0x00007f687cf68290]
Event: 883.131 Thread 0x00007f6873c9a800 14995       3       java.lang.Class::reflectionData (43 bytes)
Event: 883.132 Thread 0x00007f6873c9a800 nmethod 14995 0x00007f687cca7090 code [0x00007f687cca7260, 0x00007f687cca7630]
Event: 883.132 Thread 0x00007f6873c9a800 14997       3       java.math.BigInteger::valueOf (62 bytes)
Event: 883.132 Thread 0x00007f6873c9a800 nmethod 14997 0x00007f687ce29590 code [0x00007f687ce29760, 0x00007f687ce29b00]
Event: 883.132 Thread 0x00007f6873c9a800 14998       3       java.math.BigInteger::toByteArray (78 bytes)
Event: 883.133 Thread 0x00007f6873c9a800 nmethod 14998 0x00007f687ce46710 code [0x00007f687ce468e0, 0x00007f687ce46c70]
Event: 883.133 Thread 0x00007f6873c9a800 14996       1       java.lang.Byte::byteValue (5 bytes)
Event: 883.133 Thread 0x00007f6873c9a800 nmethod 14996 0x00007f68844a6f90 code [0x00007f68844a7140, 0x00007f68844a71f0]
Event: 883.134 Thread 0x00007f6873c9a800 15000       3       java.math.MutableBigInteger::getMagnitudeArray (45 bytes)
Event: 883.134 Thread 0x00007f6873c9a800 nmethod 15000 0x00007f687ceaec90 code [0x00007f687ceaee60, 0x00007f687ceaf020]
Event: 883.134 Thread 0x00007f6873c9a800 15001       3       sun.security.util.math.intpoly.IntegerPolynomial::get0 (5 bytes)
Event: 883.134 Thread 0x00007f6873c9a800 nmethod 15001 0x00007f687d39e810 code [0x00007f687d39e9e0, 0x00007f687d39ed20]
Event: 883.135 Thread 0x00007f6873c9a800 15002       1       sun.security.util.math.intpoly.IntegerPolynomial::getSize (5 bytes)
Event: 883.136 Thread 0x00007f6873c9a800 nmethod 15002 0x00007f68844a7290 code [0x00007f68844a7440, 0x00007f68844a74f0]
Event: 883.136 Thread 0x00007f6873c9a800 15003       3       sun.security.provider.ByteArrayAccess::b2lBig128 (350 bytes)
Event: 883.137 Thread 0x00007f6873c9a800 nmethod 15003 0x00007f687cb04890 code [0x00007f687cb04b60, 0x00007f687cb05480]

GC Heap History (20 events):
Event: 881.126 GC heap before
{Heap before GC invocations=571 (full 164):
 garbage-first heap   total 2097152K, used 2093601K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 1 young (1024K), 0 survivors (0K)
 Metaspace       used 203971K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 881.134 GC heap after
{Heap after GC invocations=572 (full 164):
 garbage-first heap   total 2097152K, used 2093601K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203971K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 881.134 GC heap before
{Heap before GC invocations=572 (full 164):
 garbage-first heap   total 2097152K, used 2093601K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203971K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 883.121 GC heap after
{Heap after GC invocations=573 (full 165):
 garbage-first heap   total 2097152K, used 2092594K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203971K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 883.140 GC heap before
{Heap before GC invocations=573 (full 165):
 garbage-first heap   total 2097152K, used 2092594K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 1 young (1024K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 883.147 GC heap after
{Heap after GC invocations=574 (full 165):
 garbage-first heap   total 2097152K, used 2093618K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 883.147 GC heap before
{Heap before GC invocations=574 (full 165):
 garbage-first heap   total 2097152K, used 2093618K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 885.130 GC heap after
{Heap after GC invocations=575 (full 166):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 885.131 GC heap before
{Heap before GC invocations=575 (full 166):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 885.135 GC heap after
{Heap after GC invocations=576 (full 166):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 885.136 GC heap before
{Heap before GC invocations=576 (full 166):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 887.136 GC heap after
{Heap after GC invocations=577 (full 167):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 887.136 GC heap before
{Heap before GC invocations=577 (full 167):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 889.321 GC heap after
{Heap after GC invocations=578 (full 168):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 889.322 GC heap before
{Heap before GC invocations=578 (full 168):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 889.326 GC heap after
{Heap after GC invocations=579 (full 168):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 889.326 GC heap before
{Heap before GC invocations=579 (full 168):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 891.547 GC heap after
{Heap after GC invocations=580 (full 169):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 891.547 GC heap before
{Heap before GC invocations=580 (full 169):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}
Event: 893.608 GC heap after
{Heap after GC invocations=581 (full 170):
 garbage-first heap   total 2097152K, used 2092646K [0x0000000080000000, 0x0000000100000000)
  region size 1024K, 0 young (0K), 0 survivors (0K)
 Metaspace       used 203976K, capacity 205929K, committed 209076K, reserved 1216512K
  class space    used 40716K, capacity 41504K, committed 41880K, reserved 1048576K
}

Deoptimization events (20 events):
Event: 787.770 Thread 0x00007f6863122800 Uncommon trap: trap_request=0xffffff4d fr.pc=0x00007f6883d22770 relative=0x00000000000005d0
Event: 787.770 Thread 0x00007f6863122800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f6883d22770 method=jdk.internal.ref.PhantomCleanable.remove()Z @ 12 c2
Event: 787.770 Thread 0x00007f6863122800 DEOPT PACKING pc=0x00007f6883d22770 sp=0x00007f6861b13d00
Event: 787.770 Thread 0x00007f6863122800 DEOPT UNPACKING pc=0x00007f687c1662a5 sp=0x00007f6861b13cb8 mode 2
Event: 789.983 Thread 0x00007f6861132000 Uncommon trap: trap_request=0xffffff4d fr.pc=0x00007f6884045af4 relative=0x0000000000000134
Event: 789.983 Thread 0x00007f6861132000 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f6884045af4 method=java.util.concurrent.ConcurrentHashMap.addCount(JI)V @ 34 c2
Event: 789.983 Thread 0x00007f6861132000 DEOPT PACKING pc=0x00007f6884045af4 sp=0x00007f685d4134b0
Event: 789.983 Thread 0x00007f6861132000 DEOPT UNPACKING pc=0x00007f687c1662a5 sp=0x00007f685d413410 mode 2
Event: 792.345 Thread 0x00007f6861121800 DEOPT PACKING pc=0x00007f687de06060 sp=0x00007f685e91cfc0
Event: 792.345 Thread 0x00007f6861121800 DEOPT UNPACKING pc=0x00007f687c1a79fa sp=0x00007f685e91c4f8 mode 0
Event: 820.051 Thread 0x00007f68682dc000 DEOPT PACKING pc=0x00007f687d760742 sp=0x00007f6865cbf190
Event: 820.051 Thread 0x00007f68682dc000 DEOPT UNPACKING pc=0x00007f687c1a79fa sp=0x00007f6865cbe608 mode 0
Event: 862.155 Thread 0x00007f6862664800 Uncommon trap: trap_request=0xffffff4d fr.pc=0x00007f6884262fb0 relative=0x0000000000000190
Event: 862.155 Thread 0x00007f6862664800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f6884262fb0 method=sun.nio.cs.Surrogate$Parser.parse(C[CII)I @ 33 c2
Event: 862.155 Thread 0x00007f6862664800 DEOPT PACKING pc=0x00007f6884262fb0 sp=0x00007f6860afa6c0
Event: 862.155 Thread 0x00007f6862664800 DEOPT UNPACKING pc=0x00007f687c1662a5 sp=0x00007f6860afa658 mode 2
Event: 862.155 Thread 0x00007f6862664800 Uncommon trap: trap_request=0xffffff4d fr.pc=0x00007f688419300c relative=0x0000000000000c6c
Event: 862.155 Thread 0x00007f6862664800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f688419300c method=sun.nio.cs.UTF_8$Encoder.encodeArrayLoop(Ljava/nio/CharBuffer;Ljava/nio/ByteBuffer;)Ljava/nio/charset/CoderResult; @ 271 c2
Event: 862.155 Thread 0x00007f6862664800 DEOPT PACKING pc=0x00007f688419300c sp=0x00007f6860afa700
Event: 862.155 Thread 0x00007f6862664800 DEOPT UNPACKING pc=0x00007f687c1662a5 sp=0x00007f6860afa6d0 mode 2

Classes redefined (0 events):
No events

Internal exceptions (20 events):
Event: 826.351 Thread 0x00007f6861121800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff32e50}> (0x00000000fff32e50) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 826.351 Thread 0x00007f6861121800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff33868}> (0x00000000fff33868) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 828.373 Thread 0x00007f6861132000 Exception <a 'java/io/IOException'{0x00000000fffaa6b0}> (0x00000000fffaa6b0) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 847.663 Thread 0x00007f6861121800 Exception <a 'java/net/SocketException'{0x00000000ffe96c58}> (0x00000000ffe96c58) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 847.663 Thread 0x00007f6861121800 Exception <a 'java/net/SocketException'{0x00000000fff72e70}> (0x00000000fff72e70) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 847.664 Thread 0x00007f6861128000 Exception <a 'java/io/IOException'{0x00000000fffbcfc0}> (0x00000000fffbcfc0) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 864.253 Thread 0x00007f68682d8800 Exception <a 'java/io/IOException'{0x00000000fff6d5d8}> (0x00000000fff6d5d8) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.124 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff14f20}> (0x00000000fff14f20) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff167a8}> (0x00000000fff167a8) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff17160}> (0x00000000fff17160) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff17b78}> (0x00000000fff17b78) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff18530}> (0x00000000fff18530) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff19ef8}> (0x00000000fff19ef8) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1a910}> (0x00000000fff1a910) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1b2c8}> (0x00000000fff1b2c8) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1bc80}> (0x00000000fff1bc80) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1c698}> (0x00000000fff1c698) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.128 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1d050}> (0x00000000fff1d050) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.129 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1da08}> (0x00000000fff1da08) thrown at [src/hotspot/share/prims/jni.cpp, line 617]
Event: 883.129 Thread 0x00007f686111f800 Exception <a 'java/io/FileNotFoundException'{0x00000000fff1e420}> (0x00000000fff1e420) thrown at [src/hotspot/share/prims/jni.cpp, line 617]

Events (20 events):
Event: 876.757 loading class com/netsuite/jdbc/oabase/oaq
Event: 876.757 loading class com/netsuite/jdbc/oabase/oaq done
Event: 876.759 Executing VM operation: G1CollectForAllocation
Event: 878.923 Executing VM operation: G1CollectForAllocation done
Event: 878.926 Executing VM operation: G1CollectForAllocation
Event: 878.926 Thread 0x00007f6860ed4800 Thread added: 0x00007f6860ed4800
Event: 878.927 Protecting memory [0x00007f685ca10000,0x00007f685ca14000] with protection modes 0
Event: 881.084 Executing VM operation: G1CollectForAllocation done
Event: 881.115 Executing VM operation: G1CollectForAllocation
Event: 881.121 Executing VM operation: G1CollectForAllocation done
Event: 881.124 Executing VM operation: G1CollectForAllocation
Event: 883.122 Executing VM operation: G1CollectForAllocation done
Event: 883.138 Executing VM operation: G1CollectForAllocation
Event: 883.139 Executing VM operation: G1CollectForAllocation done
Event: 883.139 Executing VM operation: G1CollectForAllocation
Event: 885.130 Executing VM operation: G1CollectForAllocation done
Event: 885.130 Executing VM operation: G1CollectForAllocation
Event: 889.321 Executing VM operation: G1CollectForAllocation done
Event: 889.321 Executing VM operation: G1CollectForAllocation
Event: 893.609 Executing VM operation: G1CollectForAllocation done


Dynamic libraries:
555adeba2000-555adeba3000 r--p 00000000 103:01 9213032                   /opt/java/openjdk/bin/java
555adeba3000-555adeba4000 r-xp 00001000 103:01 9213032                   /opt/java/openjdk/bin/java
555adeba4000-555adeba5000 r--p 00002000 103:01 9213032                   /opt/java/openjdk/bin/java
555adeba5000-555adeba6000 r--p 00002000 103:01 9213032                   /opt/java/openjdk/bin/java
555adeba6000-555adeba7000 rw-p 00003000 103:01 9213032                   /opt/java/openjdk/bin/java
...
7f686e876000-7f686e880000 r--p 00000000 103:01 312844                    /opt/java/openjdk/lib/libsunec.so
7f686e880000-7f686e8aa000 r-xp 0000a000 103:01 312844                    /opt/java/openjdk/lib/libsunec.so
7f686e8aa000-7f686e8b9000 r--p 00034000 103:01 312844                    /opt/java/openjdk/lib/libsunec.so
7f686e8b9000-7f686e8be000 r--p 00042000 103:01 312844                    /opt/java/openjdk/lib/libsunec.so
7f686e8be000-7f686e8c0000 rw-p 00047000 103:01 312844                    /opt/java/openjdk/lib/libsunec.so
7f68724cf000-7f68724d1000 r--p 00000000 103:01 312837                    /opt/java/openjdk/lib/libmanagement_ext.so
7f68724d1000-7f68724d4000 r-xp 00002000 103:01 312837                    /opt/java/openjdk/lib/libmanagement_ext.so
7f68724d4000-7f68724d6000 r--p 00005000 103:01 312837                    /opt/java/openjdk/lib/libmanagement_ext.so
7f68724d6000-7f68724d7000 r--p 00006000 103:01 312837                    /opt/java/openjdk/lib/libmanagement_ext.so
7f68724d7000-7f68724d8000 rw-p 00007000 103:01 312837                    /opt/java/openjdk/lib/libmanagement_ext.so
7f68726ff000-7f6872702000 r--p 00000000 103:01 312835                    /opt/java/openjdk/lib/libmanagement.so
7f6872702000-7f6872703000 r-xp 00003000 103:01 312835                    /opt/java/openjdk/lib/libmanagement.so
7f6872703000-7f6872704000 r--p 00004000 103:01 312835                    /opt/java/openjdk/lib/libmanagement.so
7f6872704000-7f6872705000 r--p 00004000 103:01 312835                    /opt/java/openjdk/lib/libmanagement.so
7f6872705000-7f6872706000 rw-p 00005000 103:01 312835                    /opt/java/openjdk/lib/libmanagement.so
7f68733fe000-7f6873402000 r--p 00000000 103:01 312839                    /opt/java/openjdk/lib/libnet.so
7f6873402000-7f6873411000 r-xp 00004000 103:01 312839                    /opt/java/openjdk/lib/libnet.so
7f6873411000-7f6873415000 r--p 00013000 103:01 312839                    /opt/java/openjdk/lib/libnet.so
7f6873415000-7f6873416000 r--p 00016000 103:01 312839                    /opt/java/openjdk/lib/libnet.so
7f6873416000-7f6873417000 rw-p 00017000 103:01 312839                    /opt/java/openjdk/lib/libnet.so
7f6873417000-7f687341d000 r--p 00000000 103:01 312840                    /opt/java/openjdk/lib/libnio.so
7f687341d000-7f6873424000 r-xp 00006000 103:01 312840                    /opt/java/openjdk/lib/libnio.so
7f6873424000-7f6873427000 r--p 0000d000 103:01 312840                    /opt/java/openjdk/lib/libnio.so
7f6873427000-7f6873428000 r--p 0000f000 103:01 312840                    /opt/java/openjdk/lib/libnio.so
7f6873428000-7f6873429000 rw-p 00010000 103:01 312840                    /opt/java/openjdk/lib/libnio.so
7f688b15e000-7f6890c96000 r--s 00000000 103:01 312848                    /opt/java/openjdk/lib/modules
7f6890c96000-7f6890c98000 r--p 00000000 103:01 312847                    /opt/java/openjdk/lib/libzip.so
7f6890c98000-7f6890c9d000 r-xp 00002000 103:01 312847                    /opt/java/openjdk/lib/libzip.so
7f6890c9d000-7f6890c9f000 r--p 00007000 103:01 312847                    /opt/java/openjdk/lib/libzip.so
7f6890c9f000-7f6890ca0000 r--p 00008000 103:01 312847                    /opt/java/openjdk/lib/libzip.so
7f6890ca0000-7f6890ca1000 rw-p 00009000 103:01 312847                    /opt/java/openjdk/lib/libzip.so
7f6890cae000-7f6890cb6000 rw-s 00000000 103:01 1163116                   /tmp/hsperfdata_metabase/1
7f6890cdf000-7f6890ce2000 r--p 00000000 103:01 312799                    /opt/java/openjdk/lib/libjimage.so
7f6890ce2000-7f6890cf5000 r-xp 00003000 103:01 312799                    /opt/java/openjdk/lib/libjimage.so
7f6890cf5000-7f6890cfb000 r--p 00016000 103:01 312799                    /opt/java/openjdk/lib/libjimage.so
7f6890cfb000-7f6890cfd000 r--p 0001b000 103:01 312799                    /opt/java/openjdk/lib/libjimage.so
7f6890cfd000-7f6890cfe000 rw-p 0001d000 103:01 312799                    /opt/java/openjdk/lib/libjimage.so
7f6890d05000-7f6890d13000 r--p 00000000 103:01 312795                    /opt/java/openjdk/lib/libjava.so
7f6890d13000-7f6890d29000 r-xp 0000e000 103:01 312795                    /opt/java/openjdk/lib/libjava.so
7f6890d29000-7f6890d30000 r--p 00024000 103:01 312795                    /opt/java/openjdk/lib/libjava.so
7f6890d30000-7f6890d31000 r--p 0002a000 103:01 312795                    /opt/java/openjdk/lib/libjava.so
7f6890d31000-7f6890d32000 rw-p 0002b000 103:01 312795                    /opt/java/openjdk/lib/libjava.so
7f6890d33000-7f6890d38000 r--p 00000000 103:01 312846                    /opt/java/openjdk/lib/libverify.so
7f6890d38000-7f6890d3f000 r-xp 00005000 103:01 312846                    /opt/java/openjdk/lib/libverify.so
7f6890d3f000-7f6890d42000 r--p 0000c000 103:01 312846                    /opt/java/openjdk/lib/libverify.so
7f6890d42000-7f6890d44000 r--p 0000e000 103:01 312846                    /opt/java/openjdk/lib/libverify.so
7f6890d44000-7f6890d45000 rw-p 00010000 103:01 312846                    /opt/java/openjdk/lib/libverify.so
7f6890e5a000-7f68910df000 r--p 00000000 103:01 312854                    /opt/java/openjdk/lib/server/libjvm.so
7f68910df000-7f6891d85000 r-xp 00285000 103:01 312854                    /opt/java/openjdk/lib/server/libjvm.so
7f6891d85000-7f6891ff4000 r--p 00f2b000 103:01 312854                    /opt/java/openjdk/lib/server/libjvm.so
7f6891ff4000-7f68920b9000 r--p 01199000 103:01 312854                    /opt/java/openjdk/lib/server/libjvm.so
7f68920b9000-7f68920f4000 rw-p 0125e000 103:01 312854                    /opt/java/openjdk/lib/server/libjvm.so
7f689214c000-7f689214f000 r--p 00000000 103:01 12590873                  /lib/libz.so.1.2.13
7f689214f000-7f689215d000 r-xp 00003000 103:01 12590873                  /lib/libz.so.1.2.13
7f689215d000-7f6892164000 r--p 00011000 103:01 12590873                  /lib/libz.so.1.2.13
7f6892164000-7f6892165000 r--p 00017000 103:01 12590873                  /lib/libz.so.1.2.13
7f6892165000-7f6892166000 rw-p 00018000 103:01 12590873                  /lib/libz.so.1.2.13
7f6892166000-7f6892168000 r--p 00000000 103:01 9213113                   /opt/java/openjdk/lib/jli/libjli.so
7f6892168000-7f6892172000 r-xp 00002000 103:01 9213113                   /opt/java/openjdk/lib/jli/libjli.so
7f6892172000-7f6892176000 r--p 0000c000 103:01 9213113                   /opt/java/openjdk/lib/jli/libjli.so
7f6892176000-7f6892177000 r--p 0000f000 103:01 9213113                   /opt/java/openjdk/lib/jli/libjli.so
7f6892177000-7f6892178000 rw-p 00010000 103:01 9213113                   /opt/java/openjdk/lib/jli/libjli.so
7f6892178000-7f689218c000 r--p 00000000 103:01 12589978                  /lib/ld-musl-x86_64.so.1
7f689218c000-7f68921d8000 r-xp 00014000 103:01 12589978                  /lib/ld-musl-x86_64.so.1
7f68921d8000-7f689220e000 r--p 00060000 103:01 12589978                  /lib/ld-musl-x86_64.so.1
7f689220e000-7f689220f000 r--p 00095000 103:01 12589978                  /lib/ld-musl-x86_64.so.1
7f689220f000-7f6892210000 rw-p 00096000 103:01 12589978                  /lib/ld-musl-x86_64.so.1


VM Arguments:
jvm_args: -Xmx2g -XX:+IgnoreUnrecognizedVMOptions -Dfile.encoding=UTF-8 -Dlogfile.path=target/log -XX:+CrashOnOutOfMemoryError 
java_command: /app/metabase.jar
java_class_path (initial): /app/metabase.jar
Launcher Type: SUN_STANDARD

[Global flags]
     intx CICompilerCount                          = 2                                         {product} {ergonomic}
     uint ConcGCThreads                            = 1                                         {product} {ergonomic}
     bool CrashOnOutOfMemoryError                  = true                                      {product} {command line}
     uint G1ConcRefinementThreads                  = 2                                         {product} {ergonomic}
   size_t G1HeapRegionSize                         = 1048576                                   {product} {ergonomic}
    uintx GCDrainStackTargetSize                   = 64                                        {product} {ergonomic}
     bool IgnoreUnrecognizedVMOptions              = true                                      {product} {command line}
   size_t InitialHeapSize                          = 65011712                                  {product} {ergonomic}
   size_t MarkStackSize                            = 4194304                                   {product} {ergonomic}
   size_t MaxHeapSize                              = 2147483648                                {product} {command line}
   size_t MaxNewSize                               = 1287651328                                {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 1048576                                   {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5825164                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122916538                              {pd product} {ergonomic}
    uintx ProfiledCodeHeapSize                     = 122916538                              {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseG1GC                                  = true                                      {product} {ergonomic}

Logging:
Log output configuration:
 #0: stdout all=warning uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/opt/java/openjdk
PATH=/opt/java/openjdk/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
LD_LIBRARY_PATH=/opt/java/openjdk/lib/server:/opt/java/openjdk/lib:/opt/java/openjdk/../lib
SHELL=/bin/sh
LANG=en_US.UTF-8
LC_ALL=en_US.UTF-8
LC_CTYPE=en_US.UTF-8

Signal Handlers:
SIGSEGV: [libjvm.so+0xeaa2d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xeaa2d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xeaa2d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xc1d110], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xc1d110], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xeaa2d0], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xc1cfb0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0xc1d750], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: [libjvm.so+0xc1d750], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGTERM: [libjvm.so+0xc1d750], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0xc1d750], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:NAME="Alpine Linux"
ID=alpine
VERSION_ID=3.18.4
PRETTY_NAME="Alpine Linux v3.18"
HOME_URL="https://alpinelinux.org/"
BUG_REPORT_URL="https://gitlab.alpinelinux.org/alpine/aports/-/issues"
uname:Linux 4.14.301-224.520.amzn2.x86_64 #1 SMP Fri Dec 9 09:57:03 UTC 2022 x86_64
OS uptime: 0 days 1:03 hours
libc:musl - unknown musl - unknown 
rlimit (soft/hard): STACK 10240k/10240k , CORE infinity/infinity , NPROC infinity/infinity , NOFILE 65536/65536 , AS infinity/infinity , CPU infinity/infinity , DATA infinity/infinity , FSIZE infinity/infinity , MEMLOCK infinity/infinity
load average:1.96 1.68 1.30

/proc/meminfo:
MemTotal:        3977000 kB
MemFree:          117000 kB
MemAvailable:     882272 kB
Buffers:            1060 kB
Cached:           953880 kB
SwapCached:            0 kB
Active:          3241860 kB
Inactive:         438404 kB
Active(anon):    2726992 kB
Inactive(anon):    15316 kB
Active(file):     514868 kB
Inactive(file):   423088 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               540 kB
Writeback:             0 kB
AnonPages:       2725452 kB
Mapped:           196440 kB
Shmem:             16892 kB
Slab:             103960 kB
SReclaimable:      64188 kB
SUnreclaim:        39772 kB
KernelStack:        4976 kB
PageTables:        11448 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     1988500 kB
Committed_AS:    3999488 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
HardwareCorrupted:     0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:       87976 kB
DirectMap2M:     2996224 kB
DirectMap1G:     1048576 kB

/sys/kernel/mm/transparent_hugepage/enabled:
always [madvise] never
/sys/kernel/mm/transparent_hugepage/defrag (defrag/compaction efforts parameter):
always defer defer+madvise [madvise] never

Process Memory:
Virtual Size: 4038124K (peak: 4049968K)
Resident Set Size: 2565280K (peak: 2577468K) (anon: 2546104K, file: 19176K, shmem: 0K)
Swapped out: 0K

/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
30945
/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
65530
/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
32768

container (cgroup) information:
container_type: cgroupv1
cpu_cpuset_cpus: 0-1
cpu_memory_nodes: 0
active_processor_count: 2
cpu_quota: no quota
cpu_period: 100000
cpu_shares: no shares
memory_limit_in_bytes: unlimited
memory_and_swap_limit_in_bytes: unlimited
memory_soft_limit_in_bytes: unlimited
memory_usage_in_bytes: 2567828 k
memory_max_usage_in_bytes: 2578944 k
kernel_memory_usage_in_bytes: 9792 k
kernel_memory_max_usage_in_bytes: unlimited
kernel_memory_limit_in_bytes: 9824 k
maximum number of tasks: unlimited
current number of tasks: 127

KVM virtualization detected
Steal ticks since vm start: 16470
Steal ticks percentage since vm start:  0.092

CPU:total 2 (initial active 2) (1 cores per cpu, 2 threads per core) family 6 model 85 stepping 7 microcode 0x5003604, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, 3dnowpref, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, adx, evex, fma
CPU Model and flags from /proc/cpuinfo:
model name	: Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc cpuid tsc_known_freq pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single pti fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves ida arat pku ospke
Online cpus:
0-1
Offline cpus:

BIOS frequency limitation:
<Not Available>
Frequency switch latency (ns):
<Not Available>
Available cpu frequencies:
<Not Available>
Current governor:
<Not Available>
Core performance/turbo boost:
<Not Available>

Memory: 4k page, physical 3977000k(116876k free), swap 0k(0k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.20.1+1) for linux-amd64-musl JRE (11.0.20.1+1), built on Aug 23 2023 15:34:01 by "" with gcc 10.3.1 20211027

END.
Warning: environ value jdk-11.0.20.1+1 for key :java-version has been overwritten with 11.0.20.1
2023-10-18 14:58:57,090 INFO metabase.util :: Maximum memory available to JVM: 2.0 GB


### Information about your Metabase installation

```JSON
- Database: Postgres
- Metabase version: 0.47.4
- Metabase hosting environment: Elastic Beanstalk
- Metabase internal database: Postgres

Severity

blocking unfolding of JSON columns

Additional context

No response
@ericcj ericcj added .Needs Triage Type:Bug Product defects labels Oct 18, 2023
@paoliniluis
Copy link
Contributor

Please assign more memory to the JVM, this is not an issue with Metabase, but rather with massive JSON parsing that can’t fit into memory

@paoliniluis paoliniluis changed the title "Allow unfolding of JSON columns" reproducibly causes OutOfMemoryError in the sync-fields step of sync database schema even with Xmx2g Make json parsing more efficient Oct 18, 2023
@paoliniluis paoliniluis changed the title Make json parsing more efficient Make json parsing more efficient on the JSON unfolding feature Oct 18, 2023
@ericcj
Copy link
Contributor Author

ericcj commented Oct 18, 2023 •
edited

i couldn't find a table where the result set of the unfolding query would be more than 16MB so crashing with the 2GB it has now seems indicative of something pathological happening. but if that's not the case maybe for the purpose of this probing metabase should add some safety like "where pg_column_size(json_column) < 4096"

@piranha
Copy link
Contributor

piranha commented Dec 7, 2023

@ericcj I wonder if you've ever identified the table causing the problem? I'm trying to reproduce this and I can't with any JSON file lying around. I even imported some 23mb file from the internets, but still no luck.

I suspect that it fails when there are a lot of keys to surface as columns, but so far I've got around 100-smth columns (not much, yeah) and it's nothing. Not sure where to dig further. :)

@ericcj
Copy link
Contributor Author

ericcj commented Dec 7, 2023

i didn't identify it. i disabled the 5 biggest tables by json column sizes but it still ran past 2GB: #34754 (comment) that size seems to indicate a problem with holding large values in memory since i doubt we have 2GB of keys. i suspect if you generate large random json data you could reproduce it, but placing some limits on the size of the json that's a candidate for analyzing might be an easier stopgap i'd be happy to upgrade and turn it back on with a limit

@ericcj
Copy link
Contributor Author

ericcj commented Mar 30, 2024

I verified this is still a problem in 0.49.2 with Xmx1500m it hits the below timeout (after using its maximum heap size). It still occurs even after setting maxResultBuffer=25percent in the JDBC string so this must not be a single very large result. It's currently working with Xmx2g but obviously it's right on the line.

[da475f41-eba8-478d-8dea-09501f56af52] 2024-03-30T13:32:31-05:00 WARN metabase.task.sync-databases Cannot sync Database Application: Timed out after 10.0 s,clojure.lang.ExceptionInfo: Timed out after 10.0 s {:message "Timed out after 10.0 s"},	at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:165),	at metabase.driver.util$can_connect_with_details_QMARK_.doInvoke(util.clj:136),	at clojure.lang.RestFn.invoke(RestFn.java:442),	at metabase.task.sync_databases$sync_and_analyze_database_STAR__BANG_$fn__106365$fn__106366.invoke(sync_databases.clj:87),	at metabase.task.sync_databases$sync_and_analyze_database_STAR__BANG_$fn__106365.invoke(sync_databases.clj:86),	at metabase.task.sync_databases$sync_and_analyze_database_STAR__BANG_.invokeStatic(sync_databases.clj:83),	at metabase.task.sync_databases$sync_and_analyze_database_STAR__BANG_.invoke(sync_databases.clj:76),	at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invokeStatic(sync_databases.clj:112),	at metabase.task.sync_databases$sync_and_analyze_database_BANG_.invoke(sync_databases.clj:100),	at metabase.task.sync_databases.SyncAndAnalyzeDatabase.execute(sync_databases.clj:117),	at org.quartz.core.JobRunShell.run(JobRunShell.java:202),	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573),Caused by: java.util.concurrent.TimeoutException: Timed out after 10.0 s,	at metabase.util.jvm$deref_with_timeout.invokeStatic(jvm.clj:287),	at metabase.util.jvm$deref_with_timeout.invoke(jvm.clj:279),	at metabase.util.jvm$do_with_timeout.invokeStatic(jvm.clj:294),	at metabase.util.jvm$do_with_timeout.invoke(jvm.clj:290),	at metabase.driver.util$can_connect_with_details_QMARK_.invokeStatic(util.clj:147),	... 11 more
[255fd9ac-17a7-4342-bbd8-1043c5e07d2a] 2024-03-30T13:32:35-05:00 INFO org.quartz.impl.jdbcjobstore.JobStoreTX ClusterManager: detected 1 failed or restarted instances.

@ixipixi ixipixi added Type:Bug Product defects Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness .Escalation and removed Type:New Feature labels May 8, 2024
@darksciencebase darksciencebase added the .Team/BackendComponents also known as BEC label May 8, 2024
@darksciencebase
Copy link
Contributor

link to the message with a script to generate a (hopefully larger) json

@piranha piranha mentioned this issue May 14, 2024
Merged
@piranha piranha added this to the 0.49.11 milestone May 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@piranha piranha

Labels
Administration/Metadata & Sync .Escalation .Performance Priority:P1 Security holes w/o exploit, crashing, setup/upgrade, login, broken common features, correctness .Team/BackendComponents also known as BEC Type:Bug Product defects
Projects
None yet
Milestone
0.49.11
Development

Successfully merging a pull request may close this issue.

json unfolding: parse incrementally instead of using cheshire metabase/metabase
5 participants
@piranha @ericcj @paoliniluis @ixipixi @darksciencebase