[04:41:47.882] src/tools/ci/cores_backtrace.sh linux /tmp/cores [04:41:48.600] dumping /tmp/cores/postgres-3-30429.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [04:41:50.323] [New LWP 30429] [04:41:50.323] [New LWP 30291] [04:41:50.323] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [04:41:51.117] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [04:41:51.912] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [04:41:53.180] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [04:41:55.641] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [04:41:55.641] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [04:41:57.581] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [04:41:58.847] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [04:42:00.483] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [04:42:01.591] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [04:42:02.387] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [04:42:03.666] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [04:42:05.340] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [04:42:06.802] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [04:42:07.750] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [04:42:09.351] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [04:42:10.794] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [04:42:11.284] [Thread debugging using libthread_db enabled] [04:42:11.284] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [04:42:12.390] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [04:42:13.842] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [04:42:14.948] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [04:42:15.590] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [04:42:16.384] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [04:42:18.754] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [04:42:19.714] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [04:42:20.508] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [04:42:21.457] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [04:42:22.410] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [04:42:24.060] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [04:42:25.166] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [04:42:25.960] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [04:42:27.401] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [04:42:28.044] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [04:42:29.638] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [04:42:30.431] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [04:42:31.705] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [04:42:32.500] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [04:42:33.611] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [04:42:34.567] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [04:42:35.839] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [04:42:36.481] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [04:42:37.433] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [04:42:38.229] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [04:42:38.738] Core was generated by `postgres: cascade: startup waiting for 000000020000000000000004 '. [04:42:38.738] Program terminated with signal SIGQUIT, Quit. [04:42:38.738] #0 0x00007f4098ffffc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.765] [Current thread is 1 (LWP 30429)] [04:42:38.873] [04:42:38.873] Thread 2 (Thread 0x7f4096996a40 (LWP 30291)): [04:42:38.873] #0 0x00007f40990c2a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.873] No symbol table info available. [04:42:38.873] #1 0x00007f40990b0e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.873] No symbol table info available. [04:42:38.873] #2 0x00007f40990b1497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.873] No symbol table info available. [04:42:38.873] #3 0x00007f40990b0cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.873] No symbol table info available. [04:42:38.873] #4 0x00007f409900ca69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.873] No symbol table info available. [04:42:38.873] #5 0x00005643ae7b7950 in RestoreArchivedFile (path=path@entry=0x7ffce412d470 "\020\330\022\344\374\177", xlogfname=xlogfname@entry=0x7ffce412d8c0 "00000002", '0' , "4", recovername=recovername@entry=0x5643af2870dc "RECOVERYXLOG", expectedSize=16777216, cleanupEnabled=) at ../src/backend/access/transam/xlogarchive.c:178 [04:42:38.873] xlogpath = "pg_wal/RECOVERYXLOG\000CV\000\000\216\345\016\257CV\000\000Q\340'\257CV\000\000I\340'\257CV\000\064H\323\022\344\374\177\000\000E\340'\257\000\000\000\000\030i\231\226@\177\000\000\b\000\000\000\000\000\000\000\304\070(\257CV\000\000\230\323\022\344\374\177\000\000\270\070(\257CV\000\000\216\345\016\257CV\000\000\300\330\022\344\374\177\000\000\227p(\257CV\000\000\230\323\022\344\374\177\000\000\225p(\257CV\000\000\030i\231\226@\177\000\000\352\357\016\257CV", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [04:42:38.873] xlogRestoreCmd = 0x5643b01eb588 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000002", '0' , "4\" \"pg_wal/RECOVERYXLOG\"" [04:42:38.873] lastRestartPointFname = "00000001", '0' , "2\000\357\016\257CV", '\000' , "\001\000\000\314v\000\000\300v\031\231@\177\000\000`w\031\231@\177\000\000\314v\000\000\002\000\000\000\000\000\000\000\060\000\000\000\020\315\000\231@\177\000\000`\314\022\344\374\177", '\000' , "W\301\025\231@\177\000\000O\301\025\231@\177\000\000\210\265\036\260CV", '\000' , "@", '\000' ... [04:42:38.873] rc = [04:42:38.873] stat_buf = {st_dev = 94848699774834, st_ino = 335544320, st_nlink = 139915421547872, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 5860069074771849984, st_size = 140724134923712, st_blksize = 94848717600136, st_blocks = 140724134923872, st_atim = {tv_sec = 140724134923456, tv_nsec = 140724134923584}, st_mtim = {tv_sec = 0, tv_nsec = 94848717600136}, st_ctim = {tv_sec = 139915422274711, tv_nsec = 0}, __glibc_reserved = {94848699786638, 140724134926544, 94848701446161}} [04:42:38.873] restartSegNo = [04:42:38.873] restartRedoPtr = 33554472 [04:42:38.873] restartTli = 1 [04:42:38.873] __func__ = "RestoreArchivedFile" [04:42:38.873] #6 0x00005643ae7c65e0 in XLogFileRead (segno=140724134926448, segno@entry=4, emode=emode@entry=13, tli=tli@entry=2, source=source@entry=XLOG_FROM_ARCHIVE, notfoundOk=notfoundOk@entry=true) at ../src/backend/access/transam/xlogrecovery.c:4206 [04:42:38.873] xlogfname = "00000002", '0' , "4\000\223W\257CV\000\000\004\000\000\t\000\000\000\000\004\000\000\t", '\000' , "-\000\000\000\000\000\000" [04:42:38.873] activitymsg = "waiting for 00000002", '0' , "4\000\177\000\000 Y͙@\177\000\000p]3\257CV\000\000m]3\257CV\000\000\330\333\022\344\374\177\000\000k]3\257CV\000" [04:42:38.873] path = "\020\330\022\344\374\177\000\000\020\330\022\344\000\000\000\000\001\000\000\000\000\000\000\000J\344\016\257CV\000\000pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000b\352\016\257\000\000\000\063P\327\022\344\374\177\000\000\b\000\000\000\000\000\000\000\304\070(\257CV\000\000\b\000\000\000\000\000\000\000I\340'\257CV\000\000\370\327\022\344\374\177\000\000E\340'\257CV\000\000\216\345\016\257CV\000\000Q\340'\257CV\000\000I\340'\257CV\000\000\370\327\022\344\374\177\000\000E\340'\257CV\000\000\030i\231\226@\177\000\000,\357\016\257CV", '\000' ... [04:42:38.873] fd = [04:42:38.873] __func__ = "XLogFileRead" [04:42:38.873] #7 0x00005643ae7c67d0 in XLogFileReadAnyTLI (segno=4, emode=emode@entry=13, source=XLOG_FROM_ANY) at ../src/backend/access/transam/xlogrecovery.c:4334 [04:42:38.873] hent = [04:42:38.873] tli = 2 [04:42:38.873] cell__state = {l = , i = } [04:42:38.873] path = "\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000@\335\022\344\374\177\000\000\360\334\022\344\374\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\304\332\022\344\374\177\000\000\230\\3\257CV\000\000q\360¯CV\000\000\340\360¯CV\000\000\240\037\000\000\377\377\002\000\247\\3\257CV\000\000\340\360¯CV\000\000\002\000\000\000\000\000\000\000\213\377\016\257CV\000\000\346\332\022\344\374\177\000\000\344\332\022\344\374\177", '\000' , "\030\000\000\000\060\000\000\000\300\332\022\344\374\177\000\000d\000\017\257CV\000\000\000\000\000\000\000\000\000\000\030\000\000\000"... [04:42:38.873] cell = [04:42:38.873] fd = [04:42:38.873] tles = 0x5643b01ebcb8 [04:42:38.873] __func__ = "XLogFileReadAnyTLI" [04:42:38.873] #8 0x00005643ae7c79cc in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211552, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211528, replayTLI=2, replayLSN=67211528, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3767 [04:42:38.873] oldSource = [04:42:38.873] startWalReceiver = false [04:42:38.873] last_fail_time = 761978342726901 [04:42:38.873] now = [04:42:38.873] streaming_reply_sent = true [04:42:38.873] __func__ = "WaitForWALToBecomeAvailable" [04:42:38.873] #9 0x00005643ae7c8501 in XLogPageRead (xlogreader=0x5643b0198ed0, targetPagePtr=67207168, reqLen=4384, targetRecPtr=67211528, readBuf=0x5643b01e7d58 "\024\321\005") at ../src/backend/access/transam/xlogrecovery.c:3348 [04:42:38.873] private = 0x5643b0198eb0 [04:42:38.873] emode = 15 [04:42:38.873] targetPageOff = 98304 [04:42:38.873] targetSegNo = 4 [04:42:38.873] r = [04:42:38.873] __func__ = "XLogPageRead" [04:42:38.873] #10 0x00005643ae7d0926 in ReadPageInternal (state=state@entry=0x5643b0198ed0, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4384) at ../src/backend/access/transam/xlogreader.c:1062 [04:42:38.873] readLen = [04:42:38.873] targetPageOff = 98304 [04:42:38.873] targetSegNo = 4 [04:42:38.873] hdr = [04:42:38.873] #11 0x00005643ae7d2581 in XLogDecodeNextRecord (state=state@entry=0x5643b0198ed0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:599 [04:42:38.873] RecPtr = 67211528 [04:42:38.873] record = [04:42:38.873] targetPagePtr = 67207168 [04:42:38.873] randAccess = false [04:42:38.873] len = [04:42:38.873] total_len = [04:42:38.873] targetRecOff = 4360 [04:42:38.873] pageHeaderSize = [04:42:38.873] assembled = false [04:42:38.873] gotheader = [04:42:38.873] readOff = [04:42:38.873] decoded = 0x0 [04:42:38.873] errormsg = 0x5643ae7c5314 "\351$\375\377\377H\213uhH\205\366t,@\366\306\au&H\215VHH\203\376\270s(H\213FHH\211\342\017\266~8H\211\306\350w3\370\377\213T$P\351+\375\377\377H\215=+\204", [04:42:38.873] #12 0x00005643ae7d3123 in XLogReadAhead (state=0x5643b0198ed0, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:975 [04:42:38.873] result = [04:42:38.873] #13 0x00005643ae7bda9b in XLogPrefetcherNextBlock (pgsr_private=94848717266176, lsn=0x5643b01fd888) at ../src/backend/access/transam/xlogprefetcher.c:496 [04:42:38.873] nonblocking = [04:42:38.873] record = [04:42:38.873] prefetcher = 0x5643b0199d00 [04:42:38.873] reader = 0x5643b0198ed0 [04:42:38.873] replaying_lsn = 67211120 [04:42:38.873] __func__ = "XLogPrefetcherNextBlock" [04:42:38.873] #14 0x00005643ae7c1e64 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:256 [04:42:38.873] No locals. [04:42:38.873] #15 lrq_complete_lsn (lsn=67211528, lrq=0x5643b01fd608) at ../src/backend/access/transam/xlogprefetcher.c:294 [04:42:38.873] No locals. [04:42:38.873] #16 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x5643b0199d00, errmsg=errmsg@entry=0x7ffce4132058) at ../src/backend/access/transam/xlogprefetcher.c:1041 [04:42:38.891] record = [04:42:38.891] replayed_up_to = 67211528 [04:42:38.891] #17 0x00005643ae7c8d2e in ReadRecord (xlogprefetcher=0x5643b0199d00, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3148 [04:42:38.891] errormsg = 0x5643aecb3037 "AWAVAUATUSH\201\354X\001" [04:42:38.891] record = [04:42:38.891] xlogreader = 0x5643b0198ed0 [04:42:38.891] private = [04:42:38.891] __func__ = "ReadRecord" [04:42:38.891] #18 0x00005643ae7cbe8f in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1829 [04:42:38.891] xtime = [04:42:38.891] ru0 = {tv = {tv_sec = 1708663142, tv_usec = 183134}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1893}, {ru_maxrss = 16432, __ru_maxrss_word = 16432}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 410, __ru_minflt_word = 410}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 10, __ru_nvcsw_word = 10}, {ru_nivcsw = 2, __ru_nivcsw_word = 2}}} [04:42:38.891] record = 0x7f40962f3e68 [04:42:38.891] reachedRecoveryTarget = false [04:42:38.891] replayTLI = 2 [04:42:38.891] __func__ = "PerformWalRecovery" [04:42:38.891] #19 0x00005643ae7b5bd9 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5733 [04:42:38.891] Insert = [04:42:38.891] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14205, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1708663140, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [04:42:38.891] wasShutdown = false [04:42:38.891] didCrash = [04:42:38.891] haveTblspcMap = false [04:42:38.891] haveBackupLabel = true [04:42:38.891] EndOfLog = [04:42:38.891] EndOfLogTLI = [04:42:38.891] newTLI = [04:42:38.891] performedWalRecovery = [04:42:38.891] endOfRecoveryInfo = [04:42:38.891] abortedRecPtr = [04:42:38.891] missingContrecPtr = [04:42:38.891] oldestActiveXID = [04:42:38.891] promoted = false [04:42:38.891] __func__ = "StartupXLOG" [04:42:38.891] #20 0x00005643aec6611f in StartupProcessMain () at ../src/backend/postmaster/startup.c:254 [04:42:38.891] No locals. [04:42:38.891] #21 0x00005643aec56498 in AuxiliaryProcessMain (auxtype=auxtype@entry=StartupProcess) at ../src/backend/postmaster/auxprocess.c:142 [04:42:38.891] __func__ = "AuxiliaryProcessMain" [04:42:38.891] #22 0x00005643aec613b7 in StartChildProcess (type=type@entry=StartupProcess) at ../src/backend/postmaster/postmaster.c:5356 [04:42:38.891] pid = [04:42:38.891] __func__ = "StartChildProcess" [04:42:38.891] #23 0x00005643aec6585f in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x5643b0197d90) at ../src/backend/postmaster/postmaster.c:1462 [04:42:38.891] opt = [04:42:38.891] status = [04:42:38.891] userDoption = [04:42:38.891] listen_addr_saved = false [04:42:38.891] output_config_variable = [04:42:38.891] __func__ = "PostmasterMain" [04:42:38.891] #24 0x00005643aea9032e in main (argc=4, argv=0x5643b0197d90) at ../src/backend/main/main.c:198 [04:42:38.891] do_check_root = [04:42:38.891] [04:42:38.891] Thread 1 (LWP 30429): [04:42:38.891] #0 0x00007f4098ffffc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.891] No symbol table info available. [04:42:38.891] #1 0x00007f40990b113a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.891] No symbol table info available. [04:42:38.891] #2 0x00007f40990c2a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [04:42:38.891] No symbol table info available.