[08:05:06.049] src/tools/ci/cores_backtrace.sh linux /tmp/cores [08:05:06.753] dumping /tmp/cores/postgres-3-24100.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [08:05:08.425] [New LWP 24100] [08:05:08.425] [New LWP 23970] [08:05:08.425] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [08:05:09.186] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [08:05:09.950] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [08:05:11.165] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [08:05:13.533] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [08:05:13.533] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [08:05:15.401] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [08:05:16.611] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [08:05:18.185] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [08:05:19.249] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [08:05:20.011] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [08:05:21.242] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [08:05:22.853] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [08:05:24.254] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [08:05:25.165] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [08:05:26.702] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [08:05:28.085] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [08:05:28.556] [Thread debugging using libthread_db enabled] [08:05:28.556] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [08:05:29.622] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [08:05:31.016] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [08:05:32.078] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [08:05:32.694] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [08:05:33.458] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [08:05:35.739] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [08:05:36.653] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [08:05:37.436] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [08:05:38.347] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [08:05:39.264] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [08:05:40.853] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [08:05:41.911] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [08:05:42.679] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [08:05:44.063] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [08:05:44.680] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [08:05:46.211] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [08:05:46.977] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [08:05:48.199] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [08:05:49.108] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [08:05:50.176] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [08:05:51.097] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [08:05:52.316] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [08:05:52.932] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [08:05:53.848] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [08:05:54.610] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [08:05:55.105] Core was generated by `postgres: cascade: startup recovering 000000020000000000000004 '. [08:05:55.105] Program terminated with signal SIGQUIT, Quit. [08:05:55.105] #0 0x00007fd65f2b2fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.133] [Current thread is 1 (LWP 24100)] [08:05:55.242] [08:05:55.242] Thread 2 (Thread 0x7fd65cc49a40 (LWP 23970)): [08:05:55.242] #0 0x00007fd65f375a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.242] No symbol table info available. [08:05:55.242] #1 0x00007fd65f363e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.242] No symbol table info available. [08:05:55.242] #2 0x00007fd65f364497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.242] No symbol table info available. [08:05:55.242] #3 0x00007fd65f363cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.242] No symbol table info available. [08:05:55.242] #4 0x00007fd65f2bfa69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.242] No symbol table info available. [08:05:55.242] #5 0x0000562443b22a01 in RestoreArchivedFile (path=path@entry=0x7ffc8c702c60 "", xlogfname=xlogfname@entry=0x7ffc8c702c20 "00000003.history", recovername=recovername@entry=0x5624445f75f0 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [08:05:55.242] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000G\334ED\000\000\000\063\240*p\214\374\177\000\000\b\000\000\000\000\000\000\000\334=_D$V\000\000\b\000\000\000\000\000\000\000Y\343^D$V\000\000H+p\214\374\177\000\000U\343^D$V\000\000s\327ED$V\000\000a\343^D$V\000\000Y\343^D$V\000\000H+p\214\374\177\000\000U\343^D$V\000\000\030\231\304\\\326\177\000\000\021\341ED$V", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [08:05:55.242] xlogRestoreCmd = 0x5624468619b8 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000003.history\" \"pg_wal/RECOVERYHISTORY\"" [08:05:55.242] lastRestartPointFname = '0' , "\000u_D$V\000\000\350&p\214\374\177\000\000\243u_D$V\000\000\030\231\304\\\326\177\000\000\317\341ED$V", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [08:05:55.242] rc = [08:05:55.242] stat_buf = {st_dev = 140558656117016, st_ino = 207303860687, st_nlink = 1953720680, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 94712623484748, st_atim = {tv_sec = 94713764239219, tv_nsec = 94713765880673}, st_mtim = {tv_sec = 3747089603738133337, tv_nsec = 140722664646296}, st_ctim = {tv_sec = 1147069269, tv_nsec = 140558656117016}, __glibc_reserved = {8, 94713765903836, 140722664646376}} [08:05:55.242] restartSegNo = [08:05:55.242] restartRedoPtr = 94713764240455 [08:05:55.242] restartTli = 0 [08:05:55.242] __func__ = "RestoreArchivedFile" [08:05:55.242] #6 0x0000562443af8e15 in existsTimeLineHistory (probeTLI=probeTLI@entry=3) at ../src/backend/access/transam/timeline.c:235 [08:05:55.242] path = '\000' , "\002\000\000\000\000\000\000\000\000\031\263C$V\000\000\001\000\000\000\000\000\000\000\006\000\000\000\r\000\000\000\220\060p\214\374\177\000\000@0p\214\374\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\242\340\372D$V\000\000 \341\372D$V\000\000\024.p\214\374\177\000\000\030\240jD$V\000\000\240\037\000\000\377\377\002\000 \341\372D$V\000\000\064.p\214\374\177\000\000'\240jD$V\000\000 \341\372D$V\000\000\002\000\000\000\000\000\000\000p\361ED$V\000\000\066.p\214\374\177\000\000\064.p\214\374\177", '\000' ... [08:05:55.242] histfname = "00000003.history", '\000' [08:05:55.242] fd = [08:05:55.242] __func__ = "existsTimeLineHistory" [08:05:55.242] #7 0x0000562443af8f00 in findNewestTimeLine (startTLI=startTLI@entry=2) at ../src/backend/access/transam/timeline.c:277 [08:05:55.242] newestTLI = 2 [08:05:55.242] probeTLI = 3 [08:05:55.242] #8 0x0000562443b2d9cc in rescanLatestTimeLine (replayTLI=replayTLI@entry=2, replayLSN=replayLSN@entry=67211528) at ../src/backend/access/transam/xlogrecovery.c:4110 [08:05:55.242] newExpectedTLEs = [08:05:55.242] found = [08:05:55.242] cell = [08:05:55.242] newtarget = [08:05:55.242] oldtarget = 2 [08:05:55.242] currentTle = 0x0 [08:05:55.242] __func__ = "rescanLatestTimeLine" [08:05:55.242] #9 0x0000562443b32b7c in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211552, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211528, replayTLI=2, replayLSN=67211528, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3671 [08:05:55.242] oldSource = XLOG_FROM_STREAM [08:05:55.242] startWalReceiver = false [08:05:55.242] last_fail_time = 763372887076479 [08:05:55.242] now = [08:05:55.242] streaming_reply_sent = true [08:05:55.242] __func__ = "WaitForWALToBecomeAvailable" [08:05:55.242] #10 0x0000562443b335b2 in XLogPageRead (xlogreader=0x56244680ef60, targetPagePtr=67207168, reqLen=4384, targetRecPtr=67211528, readBuf=0x56244685e188 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3348 [08:05:55.242] private = 0x56244680ef40 [08:05:55.242] emode = 15 [08:05:55.242] targetPageOff = 98304 [08:05:55.242] targetSegNo = 4 [08:05:55.242] r = [08:05:55.242] __func__ = "XLogPageRead" [08:05:55.242] #11 0x0000562443b3b9d7 in ReadPageInternal (state=state@entry=0x56244680ef60, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4384) at ../src/backend/access/transam/xlogreader.c:1060 [08:05:55.242] readLen = [08:05:55.242] targetPageOff = 98304 [08:05:55.242] targetSegNo = 4 [08:05:55.242] hdr = [08:05:55.242] #12 0x0000562443b3d632 in XLogDecodeNextRecord (state=state@entry=0x56244680ef60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [08:05:55.242] RecPtr = 67211528 [08:05:55.242] record = [08:05:55.242] targetPagePtr = 67207168 [08:05:55.242] randAccess = false [08:05:55.242] len = [08:05:55.242] total_len = [08:05:55.242] targetRecOff = 4360 [08:05:55.242] pageHeaderSize = [08:05:55.242] assembled = false [08:05:55.242] gotheader = [08:05:55.242] readOff = [08:05:55.242] decoded = 0x0 [08:05:55.242] errormsg = 0x5624446985a7 "LOG" [08:05:55.242] #13 0x0000562443b3e1d4 in XLogReadAhead (state=0x56244680ef60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [08:05:55.242] result = [08:05:55.242] #14 0x0000562443b28b4c in XLogPrefetcherNextBlock (pgsr_private=94713801670032, lsn=0x562446873cb8) at ../src/backend/access/transam/xlogprefetcher.c:492 [08:05:55.242] nonblocking = [08:05:55.242] record = [08:05:55.242] prefetcher = 0x56244680fd90 [08:05:55.242] reader = 0x56244680ef60 [08:05:55.242] replaying_lsn = 67211120 [08:05:55.242] __func__ = "XLogPrefetcherNextBlock" [08:05:55.242] #15 0x0000562443b2cf15 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [08:05:55.242] No locals. [08:05:55.242] #16 lrq_complete_lsn (lsn=0, lrq=0x562446873a38) at ../src/backend/access/transam/xlogprefetcher.c:290 [08:05:55.242] No locals. [08:05:55.242] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x56244680fd90, errmsg=errmsg@entry=0x7ffc8c7073a8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [08:05:55.242] record = [08:05:55.242] replayed_up_to = 0 [08:05:55.242] #18 0x0000562443b33ddf in ReadRecord (xlogprefetcher=0x56244680fd90, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3148 [08:05:55.242] errormsg = 0x56244680f770 "" [08:05:55.242] record = [08:05:55.242] xlogreader = 0x56244680ef60 [08:05:55.242] private = [08:05:55.242] __func__ = "ReadRecord" [08:05:55.242] #19 0x0000562443b36f40 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1829 [08:05:55.242] xtime = [08:05:55.242] ru0 = {tv = {tv_sec = 1710057686, tv_usec = 570823}, ru = {ru_utime = {tv_sec = 0, tv_usec = 2061}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16480, __ru_maxrss_word = 16480}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 414, __ru_minflt_word = 414}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 16, __ru_nvcsw_word = 16}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [08:05:55.242] record = 0x7fd65c51ce68 [08:05:55.242] reachedRecoveryTarget = false [08:05:55.242] replayTLI = 2 [08:05:55.242] __func__ = "PerformWalRecovery" [08:05:55.261] #20 0x0000562443b20c23 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [08:05:55.261] Insert = [08:05:55.261] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14205, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1710057685, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [08:05:55.261] wasShutdown = false [08:05:55.261] didCrash = [08:05:55.261] haveTblspcMap = false [08:05:55.261] haveBackupLabel = true [08:05:55.261] EndOfLog = [08:05:55.261] EndOfLogTLI = [08:05:55.261] newTLI = [08:05:55.261] performedWalRecovery = [08:05:55.261] endOfRecoveryInfo = [08:05:55.261] abortedRecPtr = [08:05:55.261] missingContrecPtr = [08:05:55.261] oldestActiveXID = [08:05:55.261] promoted = false [08:05:55.261] __func__ = "StartupXLOG" [08:05:55.261] #21 0x0000562443fd13b9 in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [08:05:55.261] No locals. [08:05:55.261] #22 0x0000562443fc186e in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [08:05:55.261] __func__ = "AuxiliaryProcessMain" [08:05:55.261] #23 0x0000562443fcc64d in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5351 [08:05:55.261] pid = [08:05:55.261] __func__ = "StartChildProcess" [08:05:55.261] #24 0x0000562443fd0af9 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x56244680de20) at ../src/backend/postmaster/postmaster.c:1460 [08:05:55.261] opt = [08:05:55.261] status = [08:05:55.261] userDoption = [08:05:55.261] listen_addr_saved = false [08:05:55.261] output_config_variable = [08:05:55.261] __func__ = "PostmasterMain" [08:05:55.261] #25 0x0000562443dfb791 in main (argc=4, argv=0x56244680de20) at ../src/backend/main/main.c:197 [08:05:55.261] do_check_root = [08:05:55.261] [08:05:55.261] Thread 1 (LWP 24100): [08:05:55.261] #0 0x00007fd65f2b2fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.261] No symbol table info available. [08:05:55.261] #1 0x00007fd65f36413a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.261] No symbol table info available. [08:05:55.261] #2 0x00007fd65f375a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [08:05:55.261] No symbol table info available.