[03:12:50.802] src/tools/ci/cores_backtrace.sh linux /tmp/cores [03:12:51.498] dumping /tmp/cores/postgres-3-20445.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [03:12:53.294] [New LWP 20445] [03:12:53.294] [New LWP 20300] [03:12:53.294] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [03:12:54.055] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [03:12:54.979] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [03:12:57.062] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [03:12:59.614] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [03:12:59.614] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [03:13:01.660] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [03:13:02.866] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [03:13:04.521] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [03:13:05.634] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [03:13:06.455] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [03:13:07.788] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [03:13:09.491] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [03:13:10.964] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [03:13:11.872] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [03:13:13.404] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [03:13:14.945] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [03:13:15.417] [Thread debugging using libthread_db enabled] [03:13:15.417] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [03:13:16.634] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [03:13:18.063] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [03:13:19.129] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [03:13:19.768] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [03:13:20.607] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [03:13:23.106] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [03:13:24.045] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [03:13:24.855] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [03:13:26.120] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [03:13:27.091] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [03:13:28.784] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [03:13:29.883] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [03:13:30.675] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [03:13:32.115] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [03:13:32.751] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [03:13:34.277] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [03:13:35.039] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [03:13:36.259] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [03:13:37.062] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [03:13:38.166] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [03:13:39.125] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [03:13:40.387] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [03:13:41.018] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [03:13:41.975] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [03:13:42.760] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [03:13:43.250] Core was generated by `postgres: cascade: startup recovering 000000020000000000000004 '. [03:13:43.250] Program terminated with signal SIGQUIT, Quit. [03:13:43.250] #0 0x00007fa955d24fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.278] [Current thread is 1 (LWP 20445)] [03:13:43.386] [03:13:43.386] Thread 2 (Thread 0x7fa9536bba40 (LWP 20300)): [03:13:43.386] #0 0x00007fa955de7a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.386] No symbol table info available. [03:13:43.386] #1 0x00007fa955dd5e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.386] No symbol table info available. [03:13:43.386] #2 0x00007fa955dd6497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.386] No symbol table info available. [03:13:43.386] #3 0x00007fa955dd5cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.386] No symbol table info available. [03:13:43.386] #4 0x00007fa955d31a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.386] No symbol table info available. [03:13:43.386] #5 0x0000558d909c8a31 in RestoreArchivedFile (path=path@entry=0x7ffd862b7120 "", xlogfname=xlogfname@entry=0x7ffd862b70e0 "00000003.history", recovername=recovername@entry=0x558d9149a470 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [03:13:43.386] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000\311!0\221\000\000\000\063`o+\206\375\177\000\000\b\000\000\000\000\000\000\000\\lI\221\215U\000\000\b\000\000\000\000\000\000\000\251\023I\221\215U\000\000\bp+\206\375\177\000\000\245\023I\221\215U\000\000\365\034\060\221\215U\000\000\261\023I\221\215U\000\000\251\023I\221\215U\000\000\bp+\206\375\177\000\000\245\023I\221\215U\000\000\030\271kS\251\177\000\000\223&0\221\215U", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [03:13:43.386] xlogRestoreCmd = 0x558d92415988 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000003.history\" \"pg_wal/RECOVERYHISTORY\"" [03:13:43.386] lastRestartPointFname = '0' , "\000\244I\221\215U\000\000\250k+\206\375\177\000\000#\244I\221\215U\000\000\030\271kS\251\177\000\000Q'0\221\215U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [03:13:43.386] rc = [03:13:43.386] stat_buf = {st_dev = 3472113003596310384, st_ino = 206966894640, st_nlink = 34184295202646376, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 8, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 94064083423052, st_atim = {tv_sec = 94066514599157, tv_nsec = 94066516235185}, st_mtim = {tv_sec = 3747088956488487849, tv_nsec = 140726854445912}, st_ctim = {tv_sec = 2437485477, tv_nsec = 140365225769240}, __glibc_reserved = {8, 94066516257884, 140726854445992}} [03:13:43.386] restartSegNo = [03:13:43.386] restartRedoPtr = 94066514600393 [03:13:43.386] restartTli = 0 [03:13:43.386] __func__ = "RestoreArchivedFile" [03:13:43.386] #6 0x0000558d9099ef5c in existsTimeLineHistory (probeTLI=probeTLI@entry=3) at ../src/backend/access/transam/timeline.c:235 [03:13:43.386] path = '\000' , "\002\000\000\000\000\000\000\000\060y\235\220\215U\000\000\001\000\000\000\000\000\000\000\006\000\000\000\r\000\000\000Pu+\206\375\177\000\000\000u+\206\375\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "bl䑍U\000\000\340l䑍U\000\000\324r+\206\375\177\000\000\030\300T\221\215U\000\000\240\037\000\000\377\377\002\000\340l䑍U\000\000\364r+\206\375\177\000\000'\300T\221\215U\000\000\340l䑍U\000\000\002\000\000\000\000\000\000\000\362\066\060\221\215U\000\000\366r+\206\375\177\000\000\364r+\206\375\177", '\000' ... [03:13:43.386] histfname = "00000003.history", '\000' [03:13:43.386] fd = [03:13:43.386] __func__ = "existsTimeLineHistory" [03:13:43.386] #7 0x0000558d9099f047 in findNewestTimeLine (startTLI=startTLI@entry=2) at ../src/backend/access/transam/timeline.c:277 [03:13:43.386] newestTLI = 2 [03:13:43.386] probeTLI = 3 [03:13:43.386] #8 0x0000558d909d39fc in rescanLatestTimeLine (replayTLI=replayTLI@entry=2, replayLSN=replayLSN@entry=67211528) at ../src/backend/access/transam/xlogrecovery.c:4110 [03:13:43.386] newExpectedTLEs = [03:13:43.386] found = [03:13:43.386] cell = [03:13:43.386] newtarget = [03:13:43.386] oldtarget = 2 [03:13:43.386] currentTle = 0x0 [03:13:43.386] __func__ = "rescanLatestTimeLine" [03:13:43.386] #9 0x0000558d909d8bac in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211552, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211528, replayTLI=2, replayLSN=67211528, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3671 [03:13:43.386] oldSource = XLOG_FROM_STREAM [03:13:43.386] startWalReceiver = false [03:13:43.386] last_fail_time = 762923405048282 [03:13:43.386] now = [03:13:43.386] streaming_reply_sent = true [03:13:43.386] __func__ = "WaitForWALToBecomeAvailable" [03:13:43.386] #10 0x0000558d909d95e2 in XLogPageRead (xlogreader=0x558d923c2f60, targetPagePtr=67207168, reqLen=4384, targetRecPtr=67211528, readBuf=0x558d92412158 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3348 [03:13:43.386] private = 0x558d923c2f40 [03:13:43.386] emode = 15 [03:13:43.386] targetPageOff = 98304 [03:13:43.386] targetSegNo = 4 [03:13:43.386] r = [03:13:43.386] __func__ = "XLogPageRead" [03:13:43.386] #11 0x0000558d909e1a07 in ReadPageInternal (state=state@entry=0x558d923c2f60, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4384) at ../src/backend/access/transam/xlogreader.c:1060 [03:13:43.386] readLen = [03:13:43.386] targetPageOff = 98304 [03:13:43.386] targetSegNo = 4 [03:13:43.386] hdr = [03:13:43.386] #12 0x0000558d909e3662 in XLogDecodeNextRecord (state=state@entry=0x558d923c2f60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [03:13:43.386] RecPtr = 67211528 [03:13:43.386] record = [03:13:43.386] targetPagePtr = 67207168 [03:13:43.386] randAccess = false [03:13:43.386] len = [03:13:43.386] total_len = [03:13:43.386] targetRecOff = 4360 [03:13:43.386] pageHeaderSize = [03:13:43.386] assembled = false [03:13:43.386] gotheader = [03:13:43.386] readOff = [03:13:43.386] decoded = 0x0 [03:13:43.386] errormsg = 0x558d9153a5a7 "LOG" [03:13:43.386] #13 0x0000558d909e4204 in XLogReadAhead (state=0x558d923c2f60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [03:13:43.386] result = [03:13:43.386] #14 0x0000558d909ceb7c in XLogPrefetcherNextBlock (pgsr_private=94066532171152, lsn=0x558d92427c88) at ../src/backend/access/transam/xlogprefetcher.c:492 [03:13:43.386] nonblocking = [03:13:43.386] record = [03:13:43.386] prefetcher = 0x558d923c3d90 [03:13:43.386] reader = 0x558d923c2f60 [03:13:43.386] replaying_lsn = 67211120 [03:13:43.386] __func__ = "XLogPrefetcherNextBlock" [03:13:43.386] #15 0x0000558d909d2f45 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [03:13:43.386] No locals. [03:13:43.386] #16 lrq_complete_lsn (lsn=0, lrq=0x558d92427a08) at ../src/backend/access/transam/xlogprefetcher.c:290 [03:13:43.386] No locals. [03:13:43.386] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x558d923c3d90, errmsg=errmsg@entry=0x7ffd862bb868) at ../src/backend/access/transam/xlogprefetcher.c:1037 [03:13:43.386] record = [03:13:43.386] replayed_up_to = 0 [03:13:43.386] #18 0x0000558d909d9e0f in ReadRecord (xlogprefetcher=0x558d923c3d90, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3148 [03:13:43.386] errormsg = 0x558d923c3770 "" [03:13:43.386] record = [03:13:43.386] xlogreader = 0x558d923c2f60 [03:13:43.386] private = [03:13:43.386] __func__ = "ReadRecord" [03:13:43.386] #19 0x0000558d909dcf70 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1829 [03:13:43.386] xtime = [03:13:43.386] ru0 = {tv = {tv_sec = 1709608204, tv_usec = 496762}, ru = {ru_utime = {tv_sec = 0, tv_usec = 0}, ru_stime = {tv_sec = 0, tv_usec = 1757}, {ru_maxrss = 16312, __ru_maxrss_word = 16312}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 408, __ru_minflt_word = 408}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [03:13:43.386] record = 0x7fa952f8ee68 [03:13:43.405] reachedRecoveryTarget = false [03:13:43.405] replayTLI = 2 [03:13:43.405] __func__ = "PerformWalRecovery" [03:13:43.405] #20 0x0000558d909c6c53 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [03:13:43.405] Insert = [03:13:43.405] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14205, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1709608203, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [03:13:43.405] wasShutdown = false [03:13:43.405] didCrash = [03:13:43.405] haveTblspcMap = false [03:13:43.405] haveBackupLabel = true [03:13:43.405] EndOfLog = [03:13:43.405] EndOfLogTLI = [03:13:43.405] newTLI = [03:13:43.405] performedWalRecovery = [03:13:43.405] endOfRecoveryInfo = [03:13:43.405] abortedRecPtr = [03:13:43.405] missingContrecPtr = [03:13:43.405] oldestActiveXID = [03:13:43.405] promoted = false [03:13:43.405] __func__ = "StartupXLOG" [03:13:43.405] #21 0x0000558d90e77033 in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [03:13:43.405] No locals. [03:13:43.405] #22 0x0000558d90e674e8 in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [03:13:43.405] __func__ = "AuxiliaryProcessMain" [03:13:43.405] #23 0x0000558d90e722c7 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5351 [03:13:43.405] pid = [03:13:43.405] __func__ = "StartChildProcess" [03:13:43.405] #24 0x0000558d90e76773 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x558d923c1e20) at ../src/backend/postmaster/postmaster.c:1460 [03:13:43.405] opt = [03:13:43.405] status = [03:13:43.405] userDoption = [03:13:43.405] listen_addr_saved = false [03:13:43.405] output_config_variable = [03:13:43.405] __func__ = "PostmasterMain" [03:13:43.405] #25 0x0000558d90ca17c2 in main (argc=4, argv=0x558d923c1e20) at ../src/backend/main/main.c:197 [03:13:43.405] do_check_root = [03:13:43.405] [03:13:43.405] Thread 1 (LWP 20445): [03:13:43.405] #0 0x00007fa955d24fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.405] No symbol table info available. [03:13:43.405] #1 0x00007fa955dd613a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.405] No symbol table info available. [03:13:43.405] #2 0x00007fa955de7a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [03:13:43.405] No symbol table info available.