[18:44:44.794] src/tools/ci/cores_backtrace.sh linux /tmp/cores [18:44:45.524] dumping /tmp/cores/postgres-3-19963.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [18:44:47.196] [New LWP 19963] [18:44:47.196] [New LWP 19740] [18:44:47.196] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [18:44:47.956] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [18:44:48.717] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [18:44:49.940] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [18:44:52.486] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [18:44:52.486] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [18:44:54.535] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [18:44:55.749] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [18:44:57.380] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [18:44:58.456] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [18:44:59.230] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [18:45:00.525] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [18:45:02.207] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [18:45:03.645] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [18:45:04.552] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [18:45:06.088] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [18:45:07.471] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [18:45:07.942] [Thread debugging using libthread_db enabled] [18:45:07.942] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [18:45:09.143] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [18:45:10.544] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [18:45:11.606] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [18:45:12.221] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [18:45:12.985] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [18:45:15.262] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [18:45:16.200] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [18:45:17.004] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [18:45:17.911] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [18:45:18.834] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [18:45:20.477] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [18:45:21.536] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [18:45:22.311] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [18:45:23.721] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [18:45:24.334] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [18:45:25.863] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [18:45:26.626] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [18:45:27.847] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [18:45:28.624] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [18:45:29.705] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [18:45:30.636] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [18:45:31.863] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [18:45:32.483] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [18:45:33.407] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [18:45:34.168] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [18:45:34.664] Core was generated by `postgres: cascade: startup recovering 000000020000000000000004 '. [18:45:34.664] Program terminated with signal SIGQUIT, Quit. [18:45:34.664] #0 0x00007effa76d3fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.697] [Current thread is 1 (LWP 19963)] [18:45:34.812] [18:45:34.812] Thread 2 (Thread 0x7effa506aa40 (LWP 19740)): [18:45:34.812] #0 0x00007effa7796a61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.812] No symbol table info available. [18:45:34.812] #1 0x00007effa7784e5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.812] No symbol table info available. [18:45:34.812] #2 0x00007effa7785497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.812] No symbol table info available. [18:45:34.812] #3 0x00007effa7784cdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.812] No symbol table info available. [18:45:34.812] #4 0x00007effa76e0a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.812] No symbol table info available. [18:45:34.812] #5 0x0000558fbd4c1414 in RestoreArchivedFile (path=path@entry=0x7ffdc76157a0 "", xlogfname=xlogfname@entry=0x7ffdc7615760 "00000003.history", recovername=recovername@entry=0x558fbdfcfa10 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [18:45:34.812] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\064\000\000\000\000\000\000\000\000{]\342\275\000\000\000\063\340Ua\307\375\177\000\000\b\000\000\000\000\000\000\000\354\301\374\275\217U\000\000\b\000\000\000\000\000\000\000\331g\374\275\217U\000\000\210Va\307\375\177\000\000\325g\374\275\217U\000\000\247X⽏U\000\000\341g\374\275\217U\000\000\331g\374\275\217U\000\000\210Va\307\375\177\000\000\325g\374\275\217U\000\000\030\251\006\245\377~\000\000Eb⽏U", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [18:45:34.812] xlogRestoreCmd = 0x558fc0284938 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/025_stuck_on_old_timeline/data/t_025_stuck_on_old_timeline_primary_data/archives/00000003.history\" \"pg_wal/RECOVERYHISTORY\"" [18:45:34.812] lastRestartPointFname = '0' , "\000\371\374\275\217U\000\000(Ra\307\375\177\000\000\303\371\374\275\217U\000\000\030\251\006\245\377~\000\000\003c⽏U", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' , "L\376"... [18:45:34.812] rc = [18:45:34.812] stat_buf = {st_dev = 3472113003596310384, st_ino = 206966894640, st_nlink = 34184295202646376, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 8, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 94072673357644, st_atim = {tv_sec = 94075854411943, tv_nsec = 94075856119777}, st_mtim = {tv_sec = 3747088965828372441, tv_nsec = 140727948497368}, st_ctim = {tv_sec = 3187435477, tv_nsec = 139636450437400}, __glibc_reserved = {8, 94075856142828, 140727948497448}} [18:45:34.812] restartSegNo = [18:45:34.812] restartRedoPtr = 94075854413179 [18:45:34.812] restartTli = 0 [18:45:34.812] __func__ = "RestoreArchivedFile" [18:45:34.812] #6 0x0000558fbd495dfd in existsTimeLineHistory (probeTLI=probeTLI@entry=3) at ../src/backend/access/transam/timeline.c:235 [18:45:34.812] path = '\000' , "\002\000\000\000\000\000\000\000\023\003M\275\217U\000\000\230\000\000\000\000\000\000\000\a\000\000\000\r\000\000\000pg_wal/00000002", '0' , "3\000\260[a\307\375\177\000\000\212f\002\251\377~\000\000\350S\233\276\217U", '\000' , " \000\000\000\000\000\000\b\021(\300\217U\000\000\006\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\060", '\000' , "\240\037\000\000\377\377\002", '\000' ... [18:45:34.812] histfname = "00000003.history", '\000' [18:45:34.812] fd = [18:45:34.812] __func__ = "existsTimeLineHistory" [18:45:34.812] #7 0x0000558fbd495ee8 in findNewestTimeLine (startTLI=startTLI@entry=2) at ../src/backend/access/transam/timeline.c:277 [18:45:34.812] newestTLI = 2 [18:45:34.812] probeTLI = 3 [18:45:34.812] #8 0x0000558fbd4cc3df in rescanLatestTimeLine (replayTLI=replayTLI@entry=2, replayLSN=replayLSN@entry=67211504) at ../src/backend/access/transam/xlogrecovery.c:4113 [18:45:34.812] newExpectedTLEs = [18:45:34.812] found = [18:45:34.812] cell = [18:45:34.812] newtarget = [18:45:34.812] oldtarget = 2 [18:45:34.812] currentTle = 0x0 [18:45:34.812] __func__ = "rescanLatestTimeLine" [18:45:34.812] #9 0x0000558fbd4d158f in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=67211528, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=67211504, replayTLI=2, replayLSN=67211504, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3674 [18:45:34.812] oldSource = XLOG_FROM_STREAM [18:45:34.812] startWalReceiver = false [18:45:34.812] last_fail_time = 766780889530124 [18:45:34.812] now = [18:45:34.812] streaming_reply_sent = true [18:45:34.812] __func__ = "WaitForWALToBecomeAvailable" [18:45:34.812] #10 0x0000558fbd4d1fc5 in XLogPageRead (xlogreader=0x558fc0231f10, targetPagePtr=67207168, reqLen=4360, targetRecPtr=67211504, readBuf=0x558fc0281108 "\025\321\006") at ../src/backend/access/transam/xlogrecovery.c:3351 [18:45:34.812] private = 0x558fc0231ef0 [18:45:34.812] emode = 15 [18:45:34.812] targetPageOff = 98304 [18:45:34.812] targetSegNo = 4 [18:45:34.812] r = [18:45:34.812] __func__ = "XLogPageRead" [18:45:34.812] #11 0x0000558fbd4da420 in ReadPageInternal (state=state@entry=0x558fc0231f10, pageptr=pageptr@entry=67207168, reqLen=reqLen@entry=4360) at ../src/backend/access/transam/xlogreader.c:1060 [18:45:34.812] readLen = [18:45:34.812] targetPageOff = 98304 [18:45:34.812] targetSegNo = 4 [18:45:34.812] hdr = [18:45:34.812] #12 0x0000558fbd4dc07b in XLogDecodeNextRecord (state=state@entry=0x558fc0231f10, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [18:45:34.812] RecPtr = 67211504 [18:45:34.812] record = [18:45:34.812] targetPagePtr = 67207168 [18:45:34.812] randAccess = false [18:45:34.812] len = [18:45:34.812] total_len = [18:45:34.812] targetRecOff = 4336 [18:45:34.812] pageHeaderSize = [18:45:34.812] assembled = false [18:45:34.812] gotheader = [18:45:34.812] readOff = [18:45:34.812] decoded = 0x0 [18:45:34.812] errormsg = 0x558fbe073581 "LOG" [18:45:34.812] #13 0x0000558fbd4dcc1d in XLogReadAhead (state=0x558fc0231f10, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [18:45:34.812] result = [18:45:34.812] #14 0x0000558fbd4c755f in XLogPrefetcherNextBlock (pgsr_private=94075892215104, lsn=0x558fc0296c38) at ../src/backend/access/transam/xlogprefetcher.c:492 [18:45:34.812] nonblocking = [18:45:34.812] record = [18:45:34.812] prefetcher = 0x558fc0232d40 [18:45:34.812] reader = 0x558fc0231f10 [18:45:34.812] replaying_lsn = 67211096 [18:45:34.812] __func__ = "XLogPrefetcherNextBlock" [18:45:34.812] #15 0x0000558fbd4cb928 in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [18:45:34.812] No locals. [18:45:34.812] #16 lrq_complete_lsn (lsn=0, lrq=0x558fc02969b8) at ../src/backend/access/transam/xlogprefetcher.c:290 [18:45:34.812] No locals. [18:45:34.812] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x558fc0232d40, errmsg=errmsg@entry=0x7ffdc7619ee8) at ../src/backend/access/transam/xlogprefetcher.c:1037 [18:45:34.812] record = [18:45:34.812] replayed_up_to = 0 [18:45:34.812] #18 0x0000558fbd4d27f2 in ReadRecord (xlogprefetcher=0x558fc0232d40, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=2) at ../src/backend/access/transam/xlogrecovery.c:3151 [18:45:34.812] errormsg = 0x558fc0232720 "" [18:45:34.812] record = [18:45:34.812] xlogreader = 0x558fc0231f10 [18:45:34.812] private = [18:45:34.812] __func__ = "ReadRecord" [18:45:34.812] #19 0x0000558fbd4d5989 in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [18:45:34.812] xtime = [18:45:34.812] ru0 = {tv = {tv_sec = 1713465688, tv_usec = 990838}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1748}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 16500, __ru_maxrss_word = 16500}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 408, __ru_minflt_word = 408}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 11, __ru_nvcsw_word = 11}, {ru_nivcsw = 0, __ru_nivcsw_word = 0}}} [18:45:34.812] record = 0x7effa493de50 [18:45:34.812] reachedRecoveryTarget = false [18:45:34.812] replayTLI = 2 [18:45:34.812] __func__ = "PerformWalRecovery" [18:45:34.831] #20 0x0000558fbd4bf3ac in StartupXLOG () at ../src/backend/access/transam/xlog.c:5806 [18:45:34.831] Insert = [18:45:34.831] checkPoint = {redo = 33554472, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 738}, nextOid = 14202, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1713465687, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 738} [18:45:34.831] wasShutdown = false [18:45:34.831] didCrash = [18:45:34.831] haveTblspcMap = false [18:45:34.831] haveBackupLabel = true [18:45:34.831] EndOfLog = [18:45:34.831] EndOfLogTLI = [18:45:34.831] newTLI = [18:45:34.831] performedWalRecovery = [18:45:34.831] endOfRecoveryInfo = [18:45:34.831] abortedRecPtr = [18:45:34.831] missingContrecPtr = [18:45:34.831] oldestActiveXID = [18:45:34.831] promoted = false [18:45:34.831] __func__ = "StartupXLOG" [18:45:34.831] #21 0x0000558fbd9849ae in StartupProcessMain (startup_data=, startup_data_len=) at ../src/backend/postmaster/startup.c:258 [18:45:34.831] No locals. [18:45:34.831] #22 0x0000558fbd97bfca in postmaster_child_launch (child_type=child_type@entry=B_STARTUP, startup_data=startup_data@entry=0x0, startup_data_len=startup_data_len@entry=0, client_sock=client_sock@entry=0x0) at ../src/backend/postmaster/launch_backend.c:265 [18:45:34.831] pid = [18:45:34.831] #23 0x0000558fbd97e043 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:3928 [18:45:34.831] pid = [18:45:34.831] __func__ = "StartChildProcess" [18:45:34.831] #24 0x0000558fbd98410d in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x558fc0230dd0) at ../src/backend/postmaster/postmaster.c:1364 [18:45:34.831] opt = [18:45:34.831] status = [18:45:34.831] userDoption = [18:45:34.831] listen_addr_saved = false [18:45:34.831] output_config_variable = [18:45:34.831] __func__ = "PostmasterMain" [18:45:34.831] #25 0x0000558fbd7a4a97 in main (argc=4, argv=0x558fc0230dd0) at ../src/backend/main/main.c:197 [18:45:34.831] do_check_root = [18:45:34.831] [18:45:34.831] Thread 1 (LWP 19963): [18:45:34.831] #0 0x00007effa76d3fc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.831] No symbol table info available. [18:45:34.831] #1 0x00007effa778513a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.831] No symbol table info available. [18:45:34.831] #2 0x00007effa7796a6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [18:45:34.831] No symbol table info available.