一套Linux x86-64上的11.2.0.2数据库在startup启动阶段遭遇了ORA-27103:internal error内部错误，其出错日志如下:

SQL> startup nomount;
ORA-27103: internal error
Linux-x86_64 Error: 2: No such file or directory
Additional information: 9404423
Additional information: 2oerr 27103
Usage: oerr facility errorFacility is identified by the prefix string in the error message.
For example, if you get ORA-7300, "ora" is the facility and "7300"
is the error.  So you should type "oerr ora 7300".If you get LCD-111, type "oerr lcd 111", and so on.================= alert.log ====================
This instance was first to mount
2011-05-02 21:49:47.009000 +08:00
Use ADRCI or Support Workbench to package the incident.
See Note 411.1 at My Oracle Support for error and packaging details.
Errors in file /s01/orabase/diag/rdbms/prod/PROD1/trace/PROD1_asmb_14386.trc:
ORA-04031: unable to allocate 393240 bytes of shared memory
("large pool","unknown object","large pool","ASM map operations hashtable")
ASMB (ospid: 14386): terminating the instance due to error 4031System state dump requested by (instance=1, osid=14386 (ASMB)), summary=[abnormal instance termination].System State dumped to trace file /s01/orabase/diag/rdbms/prod/PROD1/trace/PROD1_diag_14346.trcDumping diagnostic data in directory=[cdmp_20110502214947], requested by
(instance=1, osid=14386 (ASMB)), summary=[abnormal instance termination].Instance terminated by ASMB, pid = 14386=============================system state dump============================PROCESS 24: ASMB----------------------------------------SO: 0x92c955c8, type: 2, owner: (nil), flag: INIT/-/-/0x00 if: 0x3 c: 0x3proc=0x92c955c8, name=process, file=ksu.h LINE:12451, pg=0(process) Oracle pid:24, ser:1, calls cur/top: 0x9288c778/0x9288c778flags : (0x6) SYSTEMflags2: (0x0),  flags3: (0x0)intr error: 0, call error: 0, sess error: 0, txn error 0intr queue: emptyksudlp FALSE at location: 0(post info) last post received: 2296 0 2last post received-location: ksl2.h LINE:2293 ID:kslpsrlast process to post me: 92c8e248 1 6last post sent: 0 0 26last post sent-location: ksa2.h LINE:282 ID:ksasndlast process posted by me: 92c8e248 1 6(latch info) wait_event=0 bits=0Process Group: DEFAULT, pseudo proc: 0x92d24ae0O/S info: user: oracle, term: UNKNOWN, ospid: 14386OSD pid info: Unix process pid: 14386, image: oracle@rh2.oracle.com (ASMB)----------------------------------------SO: 0x92e80a58, type: 4, owner: 0x92c955c8, flag: INIT/-/-/0x00 if: 0x3 c: 0x3proc=0x92c955c8, name=session, file=ksu.h LINE:12459, pg=0(session) sid: 13 ser: 1 trans: (nil), creator: 0x92c955c8flags: (0x51) USR/- flags_idl: (0x1) BSY/-/-/-/-/-flags2: (0x408) -/-DID: , short-term DID:txn branch: (nil)oct: 0, prv: 0, sql: (nil), psql: (nil), user: 0/SYSksuxds FALSE at location: 0service name: SYS$BACKGROUNDCurrent Wait Stack:Not in wait; last wait ended 1.501399 sec agoWait State:fixed_waits=0 flags=0x21 boundary=(nil)/-1Session Wait History:elapsed time of 1.501469 sec since last wait0: waited for 'SGA: allocation forcing component growth'=0x0, =0x0, =0x0wait_id=14 seq_num=27 snap_id=7wait times: snap=0.000000 sec, exc=0.305374 sec, total=0.305390 secwait times: max=infinitewait counts: calls=6 os=6occurred after 0.000000 sec of elapsed time1: waited for 'SGA: allocation forcing component growth'=0x0, =0x0, =0x0wait_id=20 seq_num=26 snap_id=1wait times: snap=0.000001 sec, exc=0.000001 sec, total=0.000001 secwait times: max=infinitewait counts: calls=1 os=1occurred after 0.000000 sec of elapsed time2: waited for 'SGA: allocation forcing component growth'=0x0, =0x0, =0x0wait_id=14 seq_num=25 snap_id=6

在没有阅读告警日志前我的第一反应可能是上次shutdown时Oracle进程没有被清理干净，导致shared memory segments一直没有释放，从而造成了以上ORA-27103错误。 不过其实这个问题告警日志里有明确的信息，即RDBMS Instance数据库实例在mount阶段asmb进程(负责db instance与asm instance的交互)试图从large pool大池中分配390k的空间，但遭遇了ORA-04031错误，如果asmb后台进程无法正常工作将直接导致db实例无法找到asm存储上的必要Extent，因此导致出出现了"ORA-27103: internal error:Linux-x86_64 Error: 2: No such file or directory"。 换而言之ORA-04031错误才是罪魁祸首，我们来是看该实例初始化的内存参数:

[oracle@rh2 dbs]$ strings spfilePROD1.ora |egrep "sga|memory|pool"
PROD1.__large_pool_size=16777216
*.memory_target=943718400
*.shared_pool_size=314572800
*.streams_pool_size=0

因为是11g的实例所以采用了automatic memory management特性管理直接设置了memory_target参数为900M，并设置了1号实例的large pool最小为16M，900M的大小对于10g的实例而言仍是绰绰有余的，但是显然在11gr2中设置memory_target为900M是不足以驱动这样一个"庞然大物"的。我们需要配置更多的内存，亦或者可以通过设置更大的large pool来解决令人郁闷的ORA-04031错误:

[oracle@rh2 dbs]$ strings spfilePROD1.ora > initPROD1.ora
[oracle@rh2 dbs]$ rm spfilePROD1.ora
[oracle@rh2 dbs]$ vi initPROD1.ora/* 修改memory_target为至少912M */*.memory_target=1200M

/* 成功启动!  */

SQL> startup ;
ORACLE instance started.Total System Global Area 1252663296 bytes
Fixed Size                  2226072 bytes
Variable Size             687868008 bytes
Database Buffers          553648128 bytes
Redo Buffers                8921088 bytes
Database mounted.
Database opened.Private Interface 'eth1:1' configured from GPnP for use as a private interconnect.[name='eth1:1', type=1, ip=169.254.236.169, mac=94-0c-6d-71-8c-c2, net=169.254.0.0/16, mask=255.255.0.0, use=haip:cluster_interconnect/62]
Public Interface 'eth0' configured from GPnP for use as a public interface.[name='eth0', type=1, ip=192.168.1.121, mac=6c-f0-49-03-5f-99, net=192.168.1.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:1' configured from GPnP for use as a public interface.[name='eth0:1', type=1, ip=192.168.1.133, mac=6c-f0-49-03-5f-99, net=192.168.1.0/24, mask=255.255.255.0, use=public/1]
Public Interface 'eth0:2' configured from GPnP for use as a public interface.[name='eth0:2', type=1, ip=192.168.1.122, mac=6c-f0-49-03-5f-99, net=192.168.1.0/24, mask=255.255.255.0, use=public/1]
Picked latch-free SCN scheme 3
2011-05-02 22:28:04.408000 +08:00
WARNING: db_recovery_file_dest is same as db_create_file_dest
Autotune of undo retention is turned on.
LICENSE_MAX_USERS = 0
SYS auditing is disabled
Starting up:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, OLAP, Data Mining
and Real Application Testing options.
Using parameter settings in server-side pfile /s01/oracle/product/11.2.0/dbhome_1/dbs/initPROD1.ora
System parameters with non-default values:processes                = 150shared_pool_size         = 304Mstreams_pool_size        = 0memory_target            = 1200Mcontrol_files            = "+DATA/prod/controlfile/current.261.747100215"control_files            = "+DATA/prod/controlfile/current.260.747100215"db_block_size            = 8192db_flash_cache_file      = "/flashcard/prod1cache.dsk"db_flash_cache_size      = 20Gcompatible               = "11.2.0.0.0"log_archive_dest_1       = "location=+DATA"cluster_database         = TRUEdb_create_file_dest      = "+DATA"db_recovery_file_dest    = "+DATA"db_recovery_file_dest_size= 40320Mthread                   = 1undo_tablespace          = "UNDOTBS1"instance_number          = 1db_domain                = ""dispatchers              = "(PROTOCOL=TCP) (SERVICE=PRODXDB)"remote_listener          = "rh-cluster-scan:1521"remote_listener          = "*.remote_login_pas"audit_file_dest          = "/s01/orabase/admin/PROD/adump"audit_trail              = "DB"db_name                  = "PROD"open_cursors             = 300diagnostic_dest          = "/s01/orabase"
Cluster communication is configured to use the following interface(s) for this instance169.254.236.169
cluster interconnect IPC version:Oracle UDP/IP (generic)
IPC Vendor 1 proto 2
2011-05-02 22:28:07.675000 +08:00
ORA-00132: syntax error or unresolved network name '*.remote_login_pas'
PMON started with pid=2, OS id=19807
PSP0 started with pid=3, OS id=19809
2011-05-02 22:28:08.754000 +08:00
VKTM started with pid=4, OS id=19811 at elevated priority
GEN0 started with pid=5, OS id=19815
VKTM running at (1)millisec precision with DBRM quantum (100)ms
DIAG started with pid=6, OS id=19817
DBRM started with pid=7, OS id=19819
PING started with pid=8, OS id=19821
ACMS started with pid=9, OS id=19823
DIA0 started with pid=10, OS id=19825
LMON started with pid=11, OS id=19827
LMD0 started with pid=12, OS id=19829
LMS0 started with pid=13, OS id=19831 at elevated priority
RMS0 started with pid=14, OS id=19835
LMHB started with pid=15, OS id=19837
MMAN started with pid=16, OS id=19839
* Load Monitor used for high load check
* New Low - High Load Threshold Range = [1920 - 2560]
LGWR started with pid=18, OS id=19843
DBW0 started with pid=17, OS id=19841
CKPT started with pid=19, OS id=19845
SMON started with pid=20, OS id=19847
RECO started with pid=21, OS id=19849
RBAL started with pid=22, OS id=19851
ASMB started with pid=23, OS id=19853
MMON started with pid=24, OS id=19855
starting up 1 dispatcher(s) for network address '(ADDRESS=(PARTIAL=YES)(PROTOCOL=TCP))'...
MMNL started with pid=25, OS id=19857
starting up 1 shared server(s) ...
lmon registered with NM - instance number 1 (internal mem no 0)
2011-05-02 22:28:09.825000 +08:00
NOTE: initiating MARK startup
Starting background process MARK
MARK started with pid=28, OS id=19866
NOTE: MARK has subscribed
Reconfiguration started (old inc 0, new inc 2)
List of instances:1 (myinst: 1)Global Resource Directory frozen
* allocate domain 0, invalid = TRUECommunication channels reestablishedMaster broadcasted resource hash value bitmapsNon-local Process blocks cleaned outLMS 0: 0 GCS shadows cancelled, 0 closed, 0 Xw survivedSet master node infoSubmitted all remote-enqueue requestsDwn-cvts replayed, VALBLKs dubiousAll grantable enqueues grantedPost SMON to start 1st pass IRSubmitted all GCS remote-cache requestsPost SMON to start 1st pass IRFix write in gcs resources
Reconfiguration complete
LCK0 started with pid=30, OS id=19872
Starting background process RSMN
RSMN started with pid=31, OS id=19874
ORACLE_BASE from environment = /s01/orabase
2011-05-02 22:28:12.112000 +08:00
ALTER DATABASE   MOUNT
This instance was first to mount
2011-05-02 22:28:13.202000 +08:00
NOTE: Loaded library: System
ALTER SYSTEM SET
local_listener='(DESCRIPTION=(ADDRESS_LIST=(ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.122)(PORT=1521))))'
SCOPE=MEMORY SID='PROD1';
SUCCESS: diskgroup DATA was mounted
NOTE: dependency between database PROD and diskgroup resource ora.DATA.dg is established
/* 也可以直接增大large_pool_size来解决上述问题 */large_pool_size=30M
memory_target=912M

本文转自maclean_007 51CTO博客，原文链接:http://blog.51cto.com/maclean/1277772