补丁不生效例子
aix 7202 11.2.0.4 rac 180417 psu
实例2 例行重启,实例1发生了意外重启
实例1告警日志
-
fri sep 16 00:34:57 2022
-
set master node info
-
submitted all remote-enqueue requests
-
dwn-cvts replayed, valblks dubious
-
all grantable enqueues granted
-
fri sep 16 00:34:59 2022
-
minact-scn: master returning as live inst:2 has inc# mismatch instinc:0 cur:8 errcnt:0
-
fri sep 16 00:36:16 2022
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms2_38732932.trc (incident=1488512):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488512/orcl1_lms2_38732932_i1488512.trc
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
fri sep 16 00:36:17 2022
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms0_19269850.trc (incident=1488496):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488496/orcl1_lms0_19269850_i1488496.trc
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
fri sep 16 00:36:18 2022
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms1_5505754.trc (incident=1488504):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488504/orcl1_lms1_5505754_i1488504.trc
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
fri sep 16 00:36:25 2022
-
dumping diagnostic data in directory=[cdmp_20220916003625], requested by (instance=1, osid=19269850 (lms0)), summary=[incident=1488496].
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms0_19269850.trc (incident=1488497):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488497/orcl1_lms0_19269850_i1488497.trc
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms2_38732932.trc (incident=1488513):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
incident details in: /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488513/orcl1_lms2_38732932_i1488513.trc
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
fri sep 16 00:36:27 2022
-
sweep [inc][1488513]: completed
-
sweep [inc][1488512]: completed
-
sweep [inc][1488504]: completed
-
sweep [inc][1488497]: completed
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms1_5505754.trc (incident=1488505):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms1_5505754.trc (incident=1488506):
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
use adrci or support workbench to package the incident.
-
see note 411.1 at my oracle support for error and packaging details.
-
errors in file /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms1_5505754.trc (incident=1488507):
lms2、lms0、lms1三个进程都突然发生ora-4030异常,内存不足,看相关trc文件
-
dump file /u01/app/oracle/diag/rdbms/orcl/orcl1/incident/incdir_1488512/orcl1_lms2_38732932_i1488512.trc
-
oracle database 11g enterprise edition release 11.2.0.4.0 - 64bit production
-
with the partitioning, real application clusters, automatic storage management, olap,
-
data mining and real application testing options
-
oracle_home = /u01/app/oracle/product/11.2.0/db_1
-
system name: aix
-
node name: db1
-
release: 2
-
version: 7
-
machine: 00fb76794c00
-
instance name: orcl1
-
redo thread mounted by this instance: 1
-
oracle process number: 15
-
unix process pid: 38732932, image: oracle@db1 (lms2)
-
-
-
*** 2022-09-16 00:36:16.929
-
*** session id:(1892.1) 2022-09-16 00:36:16.929
-
*** client id:() 2022-09-16 00:36:16.929
-
*** service name:(sys$background) 2022-09-16 00:36:16.929
-
*** module name:() 2022-09-16 00:36:16.929
-
*** action name:() 2022-09-16 00:36:16.929
-
-
dump continued from file: /u01/app/oracle/diag/rdbms/orcl/orcl1/trace/orcl1_lms2_38732932.trc
-
ora-04030: out of process memory when trying to allocate 195248 bytes (kjc pga smbuf,kjc pga send buffer)
-
-
========= dump for incident 1488512 (ora 4030) ========
-
----- beginning of customized incident dump(s) -----
-
=======================================
-
top 10 memory uses for this process
-
---------------------------------------
-
-
*** 2022-09-16 00:36:22.939
-
99% 31 gb, 172610 chunks: "kjc pga send buffer " <--------------这里很异常
-
kjc pga smbuf ds=110733230 dsprt=110005120
-
1% 213 mb, 172724 chunks: "free memory "
-
pga heap ds=110005120 dsprt=0
-
0% 149 mb, 145 chunks: "permanent memory "
-
pga heap ds=110005120 dsprt=0
-
0% 75 kb, 3 chunks: "permanent memory "
-
session heap ds=1109d66e0 dsprt=11011f5d8
-
0% 54 kb, 1 chunk : "free memory "
-
top call heap ds=11011f3b8 dsprt=0
-
0% 38 kb, 1 chunk : "fixed uga "
-
fixed uga heap ds=1108bad28 dsprt=110005120
-
0% 20 kb, 2 chunks: "free memory "
-
session heap ds=1109d66e0 dsprt=11011f5d8
-
0% 16 kb, 7 chunks: "permanent memory "
-
diag pga ds=1107324d0 dsprt=110005120
-
0% 16 kb, 2 chunks: "dbgeinitprocessctx:invctx "
-
diag pga ds=1107324d0 dsprt=110005120
-
0% 16 kb, 1 chunk : "kgh stack "
-
pga heap ds=110005120 dsprt=0
-
-
=======================================
-
private memory summary for this process
-
---------------------------------------
-
******************************************************
-
private heap summary dump
-
32 gb total:
-
31 gb commented, 149 mb permanent
-
213 mb free (0 kb in empty extents),
-
32 gb, 1 heap: "kjc pga smbuf " 212 mb free held
-
------------------------------------------------------
-
summary of subheaps at depth 1
-
31 gb total:
-
31 gb commented, 98 kb permanent
-
31 kb free (0 kb in empty extents),
-
31 gb, 172610 chunks: "kjc pga send buffer "
-
-
*** 2022-09-16 00:36:23.380
-
-
=========================================
-
real-free allocator dump for this process
-
-----------------------------------------
-
-
dump of real-free memory allocator heap [0x1109a4810]
-
mag=0xfefe0001 flg=0x5000003 fds=0x0 blksz=65536
-
blkdstbl=0x1109a4820, iniblk=521216 maxblk=524288 numsegs=320
-
in-use num=172723 siz=4028628992, freeable num=1 siz=65536, free num=310 siz=20316160
-
-
==========================================
-
instance-wide private memory usage summary
-
------------------------------------------
-
-
dumping work area table (level=1)
-
=====================================
-
-
global sga info
-
---------------
-
-
global target: 115507 mb
-
auto target: 11333 mb
-
max pga: 2048 mb
-
pga limit: 4096 mb
-
pga limit known: 0
-
pga limit errors: 0
-
-
pga inuse: 102955 mb
-
pga alloc: 105793 mb
-
pga freeable: 742 mb
-
pga freed: 43967956 mb
-
pga to free: 0
-
broker request: 0
-
-
pga auto: 41 mb
-
pga manual: 0 mb
-
-
pga alloc (max): 105793 mb
-
pga auto (max): 5444 mb
-
pga manual (max): 1 mb
-
-
# workareas : 43
-
# workareas(max): 224
-
-
-
-
================================
-
per-process private memory usage
-
--------------------------------
-
-
private memory usage per oracle process
-
-
-------------------------
-
top 10 processes:
-
-------------------------
-
(percentage is of 103 gb total allocated memory)
-
31% pid 13: 32 gb used of 32 gb allocated (128 kb freeable)
-
31% pid 14: 32 gb used of 32 gb allocated
-
31% pid 15: 32 gb used of 32 gb allocated (64 kb freeable) <= current proc
-
0% pid 12: 14 mb used of 113 mb allocated (98 mb freeable)
-
0% pid 10: 45 mb used of 56 mb allocated (7680 kb freeable)
-
0% pid 86: 42 mb used of 45 mb allocated
-
0% pid 87: 42 mb used of 45 mb allocated
-
0% pid 89: 42 mb used of 45 mb allocated
-
0% pid 973: 42 mb used of 43 mb allocated (1024 kb freeable)
-
0% pid 1029: 42 mb used of 43 mb allocated (1024 kb freeable)
-
-
-------------------------
-
all processes:
-
-------------------------
-
(session detail when over 5290 mb allocated)
-
-
pid 2: 2119 kb used of 2350 kb allocated
-
pid 3: 1060 kb used of 1182 kb allocated
-
pid 4: 1056 kb used of 1182 kb allocated
-
pid 5: 1066 kb used of 1182 kb allocated
-
pid 6: 6161 kb used of 6494 kb allocated (128 kb freeable)
-
pid 7: 2121 kb used of 2718 kb allocated (320 kb freeable)
-
...
三个进程每个都32g,搜索mos,发现有个bug很像
检查当前的补丁,opatch lsinv|grep 18316692 已经包含了此修复,但是lms的pga还是很大,看来没有解决。
怎么办呢?
监控,发现内存占用太大就重启实例(lms进程杀不得)
-
select pid,spid,program,pga_used_mem,pga_alloc_mem,pga_freeable_mem,pga_max_mem from v$process where program like '%lms%';
lms进程的工作机制
参考:
-
https://orainternals.files.wordpress.com/2008/03/riyaj_battle_of_nodes_rac_perf_myths_ppt.pdf
阅读(976) | 评论(0) | 转发(0) |