[07:40:27.313] src/tools/ci/cores_backtrace.sh linux /tmp/cores [07:40:28.005] dumping /tmp/cores/postgres-3-32873.core for /tmp/cirrus-ci-build/build/tmp_install/usr/local/pgsql/bin/postgres [07:40:29.670] [New LWP 32873] [07:40:29.670] [New LWP 31790] [07:40:29.670] Downloading separate debug info for /lib/x86_64-linux-gnu/libm.so.6... [07:40:30.432] Downloading separate debug info for /lib/x86_64-linux-gnu/libdl.so.2... [07:40:31.353] Downloading separate debug info for /lib/x86_64-linux-gnu/librt.so.1... [07:40:32.568] Downloading separate debug info for /lib/x86_64-linux-gnu/libgssapi_krb5.so.2... [07:40:36.060] Downloading separate debug info for /lib/x86_64-linux-gnu/libicuuc.so.67... [07:40:36.060] Downloading separate debug info for /.cache/debuginfod_client/f871bbd529a02abd860f0d16b842b5b20234cb49/debuginfo... [07:40:38.118] Downloading separate debug info for /lib/x86_64-linux-gnu/libicui18n.so.67... [07:40:39.329] Downloading separate debug info for /lib/x86_64-linux-gnu/libldap_r-2.4.so.2... [07:40:41.003] Downloading separate debug info for /lib/x86_64-linux-gnu/libxml2.so.2... [07:40:42.069] Downloading separate debug info for /lib/x86_64-linux-gnu/liblz4.so.1... [07:40:42.871] Downloading separate debug info for /lib/x86_64-linux-gnu/libpam.so.0... [07:40:44.099] Downloading separate debug info for /lib/x86_64-linux-gnu/libssl.so.1.1... [07:40:45.710] Downloading separate debug info for /lib/x86_64-linux-gnu/libcrypto.so.1.1... [07:40:47.207] Downloading separate debug info for /lib/x86_64-linux-gnu/libsystemd.so.0... [07:40:48.119] Downloading separate debug info for /lib/x86_64-linux-gnu/libz.so.1... [07:40:49.655] Downloading separate debug info for /lib/x86_64-linux-gnu/libzstd.so.1... [07:40:51.038] Downloading separate debug info for /lib/x86_64-linux-gnu/libpthread.so.0... [07:40:51.509] [Thread debugging using libthread_db enabled] [07:40:51.509] Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1". [07:40:52.575] Downloading separate debug info for /lib64/ld-linux-x86-64.so.2... [07:40:53.968] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5.so.3... [07:40:55.030] Downloading separate debug info for /lib/x86_64-linux-gnu/libk5crypto.so.3... [07:40:55.644] Downloading separate debug info for /lib/x86_64-linux-gnu/libcom_err.so.2... [07:40:56.408] Downloading separate debug info for /lib/x86_64-linux-gnu/libkrb5support.so.0... [07:40:58.690] Downloading separate debug info for /lib/x86_64-linux-gnu/libstdc++.so.6... [07:40:59.616] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcc_s.so.1... [07:41:00.381] Downloading separate debug info for /lib/x86_64-linux-gnu/liblber-2.4.so.2... [07:41:01.293] Downloading separate debug info for /lib/x86_64-linux-gnu/libresolv.so.2... [07:41:02.203] Downloading separate debug info for /lib/x86_64-linux-gnu/libsasl2.so.2... [07:41:03.798] Downloading separate debug info for /lib/x86_64-linux-gnu/libgnutls.so.30... [07:41:04.916] Downloading separate debug info for /lib/x86_64-linux-gnu/liblzma.so.5... [07:41:05.712] Downloading separate debug info for /lib/x86_64-linux-gnu/libaudit.so.1... [07:41:07.154] Downloading separate debug info for /lib/x86_64-linux-gnu/libgcrypt.so.20... [07:41:07.768] Downloading separate debug info for /lib/x86_64-linux-gnu/libkeyutils.so.1... [07:41:09.297] Downloading separate debug info for /lib/x86_64-linux-gnu/libp11-kit.so.0... [07:41:10.059] Downloading separate debug info for /lib/x86_64-linux-gnu/libidn2.so.0... [07:41:11.287] Downloading separate debug info for /lib/x86_64-linux-gnu/libunistring.so.2... [07:41:12.056] Downloading separate debug info for /lib/x86_64-linux-gnu/libtasn1.so.6... [07:41:13.128] Downloading separate debug info for /lib/x86_64-linux-gnu/libnettle.so.8... [07:41:14.051] Downloading separate debug info for /lib/x86_64-linux-gnu/libhogweed.so.6... [07:41:15.284] Downloading separate debug info for /lib/x86_64-linux-gnu/libgmp.so.10... [07:41:15.934] Downloading separate debug info for /lib/x86_64-linux-gnu/libcap-ng.so.0... [07:41:16.910] Downloading separate debug info for /lib/x86_64-linux-gnu/libgpg-error.so.0... [07:41:17.685] Downloading separate debug info for /lib/x86_64-linux-gnu/libffi.so.7... [07:41:18.183] Core was generated by `postgres: standby2: startup recovering 000000010000000000000006 '. [07:41:18.183] Program terminated with signal SIGQUIT, Quit. [07:41:18.183] #0 0x00007fb3b8c9afc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.210] [Current thread is 1 (LWP 32873)] [07:41:18.323] [07:41:18.323] Thread 2 (Thread 0x7fb3b6631a40 (LWP 31790)): [07:41:18.323] #0 0x00007fb3b8d5da61 in clone () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.323] No symbol table info available. [07:41:18.323] #1 0x00007fb3b8d4be5e in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.323] No symbol table info available. [07:41:18.323] #2 0x00007fb3b8d4c497 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.323] No symbol table info available. [07:41:18.323] #3 0x00007fb3b8d4bcdb in posix_spawn () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.323] No symbol table info available. [07:41:18.323] #4 0x00007fb3b8ca7a69 in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.323] No symbol table info available. [07:41:18.323] #5 0x000056554bc68606 in RestoreArchivedFile (path=path@entry=0x7ffc7e7e3610 "", xlogfname=xlogfname@entry=0x7ffc7e7e35d0 "00000002.history", recovername=recovername@entry=0x56554c73d650 "RECOVERYHISTORY", expectedSize=expectedSize@entry=0, cleanupEnabled=cleanupEnabled@entry=false) at ../src/backend/access/transam/xlogarchive.c:177 [07:41:18.323] xlogpath = "pg_wal/RECOVERYHISTORY\000\000\b\000\000\000\000\000\000\066\000\000\000\000\000\000\000\000\242CZL\000\000\000\062\000\000\000\000\000\000\000\000\b\000\000\000\000\000\000\000\034\236sLUV\000\000\b\000\000\000\000\000\000\000\231CsLUV\000\000\370\064~~\374\177\000\000\225CsLUV\000\000\316>ZLUV\000\000\241CsLUV\000\000\231CsLUV\000\000\370\064~~\374\177\000\000\225CsLUV\000\000\030\031c\266\263\177\000\000lHZLUV", '\000' , "\b", '\000' , "\060", '\000' , "\002\000\000\000\000\000\000\000"... [07:41:18.323] xlogRestoreCmd = 0x56554f016638 "cp \"/tmp/cirrus-ci-build/build/testrun/recovery/040_standby_failover_slots_sync/data/t_040_standby_failover_slots_sync_publisher_data/archives/00000002.history\" \"pg_wal/RECOVERYHISTORY\"" [07:41:18.323] lastRestartPointFname = '0' , "\000\326sLUV\000\000\230\060~~\374\177\000\000\003\326sLUV\000\000\030\031c\266\263\177\000\000*IZLUV", '\000' , "\b", '\000' , "\060", '\000' , "\002", '\000' ... [07:41:18.323] rc = [07:41:18.323] stat_buf = {st_dev = 140409835821336, st_ino = 207439415594, st_nlink = 0, st_mode = 0, st_uid = 0, st_gid = 0, __pad0 = 0, st_rdev = 48, st_size = 5922071200885860208, st_blksize = 6366210006019162949, st_blocks = 4673356, st_atim = {tv_sec = 0, tv_nsec = 0}, st_mtim = {tv_sec = 3891110078048108544, tv_nsec = 0}, st_ctim = {tv_sec = 0, tv_nsec = 0}, __glibc_reserved = {8, 94924354854428, 140722430685336}} [07:41:18.323] restartSegNo = [07:41:18.323] restartRedoPtr = 94924353192866 [07:41:18.323] restartTli = 0 [07:41:18.323] __func__ = "RestoreArchivedFile" [07:41:18.323] #6 0x000056554bc3ea05 in existsTimeLineHistory (probeTLI=probeTLI@entry=2) at ../src/backend/access/transam/timeline.c:235 [07:41:18.323] path = '\000' , "\001\000\000\000\000\000\000\000\005u\307KUV\000\000\300\071~~\374\177\000\000\006\000\000\000\r\000\000\000\001\000\000\000\000\000\000\000\001\000\000\000\000\000\000\000@:~~\374\177\000\000\360\071~~\374\177\000\000\001\000\000\000\000\000\000\000\004\000\000\t", '\000' , "\304\067~~\374\177\000\000\330\001\177LUV\000\000\261V\017MUV\000\000 W\017MUV\000\000\240\037\000\000\377\377\002\000\347\001\177LUV\000\000 W\017MUV\000\000\002\000\000\000\000\000\000\000\313XZLUV\000\000\346\067~~\374\177\000\000\344\067~~\374\177", '\000' ... [07:41:18.323] histfname = "00000002.history", '\000' [07:41:18.323] fd = [07:41:18.323] __func__ = "existsTimeLineHistory" [07:41:18.323] #7 0x000056554bc3eaf0 in findNewestTimeLine (startTLI=startTLI@entry=1) at ../src/backend/access/transam/timeline.c:277 [07:41:18.323] newestTLI = 1 [07:41:18.323] probeTLI = 2 [07:41:18.323] #8 0x000056554bc735d1 in rescanLatestTimeLine (replayTLI=replayTLI@entry=1, replayLSN=replayLSN@entry=100683536) at ../src/backend/access/transam/xlogrecovery.c:4113 [07:41:18.323] newExpectedTLEs = [07:41:18.323] found = [07:41:18.323] cell = [07:41:18.323] newtarget = [07:41:18.323] oldtarget = 1 [07:41:18.323] currentTle = 0x0 [07:41:18.323] __func__ = "rescanLatestTimeLine" [07:41:18.323] #9 0x000056554bc78781 in WaitForWALToBecomeAvailable (RecPtr=RecPtr@entry=100683560, randAccess=false, fetching_ckpt=false, tliRecPtr=tliRecPtr@entry=100683536, replayTLI=1, replayLSN=100683536, nonblocking=false) at ../src/backend/access/transam/xlogrecovery.c:3674 [07:41:18.323] oldSource = XLOG_FROM_STREAM [07:41:18.323] startWalReceiver = false [07:41:18.323] last_fail_time = 763976290926469 [07:41:18.323] now = [07:41:18.323] streaming_reply_sent = true [07:41:18.323] __func__ = "WaitForWALToBecomeAvailable" [07:41:18.323] #10 0x000056554bc791b7 in XLogPageRead (xlogreader=0x56554efc3f60, targetPagePtr=100679680, reqLen=3880, targetRecPtr=100683536, readBuf=0x56554f012e08 "\024\321\006") at ../src/backend/access/transam/xlogrecovery.c:3351 [07:41:18.323] private = 0x56554efc3f40 [07:41:18.323] emode = 15 [07:41:18.323] targetPageOff = 16384 [07:41:18.323] targetSegNo = 6 [07:41:18.323] r = [07:41:18.323] __func__ = "XLogPageRead" [07:41:18.323] #11 0x000056554bc81612 in ReadPageInternal (state=state@entry=0x56554efc3f60, pageptr=pageptr@entry=100679680, reqLen=reqLen@entry=3880) at ../src/backend/access/transam/xlogreader.c:1060 [07:41:18.323] readLen = [07:41:18.323] targetPageOff = 16384 [07:41:18.323] targetSegNo = 6 [07:41:18.323] hdr = [07:41:18.323] #12 0x000056554bc8326d in XLogDecodeNextRecord (state=state@entry=0x56554efc3f60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:597 [07:41:18.323] RecPtr = 100683536 [07:41:18.323] record = [07:41:18.323] targetPagePtr = 100679680 [07:41:18.323] randAccess = false [07:41:18.323] len = [07:41:18.323] total_len = [07:41:18.323] targetRecOff = 3856 [07:41:18.323] pageHeaderSize = [07:41:18.323] assembled = false [07:41:18.323] gotheader = [07:41:18.323] readOff = [07:41:18.323] decoded = 0x0 [07:41:18.323] errormsg = 0x56554c7de747 "LOG" [07:41:18.323] #13 0x000056554bc83e0f in XLogReadAhead (state=0x56554efc3f60, nonblocking=nonblocking@entry=false) at ../src/backend/access/transam/xlogreader.c:973 [07:41:18.323] result = [07:41:18.323] #14 0x000056554bc6e751 in XLogPrefetcherNextBlock (pgsr_private=94924397366672, lsn=0x56554f028788) at ../src/backend/access/transam/xlogprefetcher.c:492 [07:41:18.323] nonblocking = [07:41:18.323] record = [07:41:18.323] prefetcher = 0x56554efc4d90 [07:41:18.323] reader = 0x56554efc3f60 [07:41:18.323] replaying_lsn = 100683488 [07:41:18.323] __func__ = "XLogPrefetcherNextBlock" [07:41:18.323] #15 0x000056554bc72b1a in lrq_prefetch (lrq=) at ../src/backend/access/transam/xlogprefetcher.c:252 [07:41:18.323] No locals. [07:41:18.323] #16 lrq_complete_lsn (lsn=0, lrq=0x56554f0286b8) at ../src/backend/access/transam/xlogprefetcher.c:290 [07:41:18.323] No locals. [07:41:18.323] #17 XLogPrefetcherReadRecord (prefetcher=prefetcher@entry=0x56554efc4d90, errmsg=errmsg@entry=0x7ffc7e7e7d58) at ../src/backend/access/transam/xlogprefetcher.c:1037 [07:41:18.323] record = [07:41:18.323] replayed_up_to = 0 [07:41:18.323] #18 0x000056554bc799e4 in ReadRecord (xlogprefetcher=0x56554efc4d90, emode=emode@entry=15, fetching_ckpt=fetching_ckpt@entry=false, replayTLI=1) at ../src/backend/access/transam/xlogrecovery.c:3151 [07:41:18.323] errormsg = 0x56554efc4770 "" [07:41:18.323] record = [07:41:18.323] xlogreader = 0x56554efc3f60 [07:41:18.323] private = [07:41:18.323] __func__ = "ReadRecord" [07:41:18.323] #19 0x000056554bc7cb7b in PerformWalRecovery () at ../src/backend/access/transam/xlogrecovery.c:1832 [07:41:18.323] xtime = [07:41:18.323] ru0 = {tv = {tv_sec = 1710661086, tv_usec = 509773}, ru = {ru_utime = {tv_sec = 0, tv_usec = 1495}, ru_stime = {tv_sec = 0, tv_usec = 0}, {ru_maxrss = 15832, __ru_maxrss_word = 15832}, {ru_ixrss = 0, __ru_ixrss_word = 0}, {ru_idrss = 0, __ru_idrss_word = 0}, {ru_isrss = 0, __ru_isrss_word = 0}, {ru_minflt = 401, __ru_minflt_word = 401}, {ru_majflt = 5, __ru_majflt_word = 5}, {ru_nswap = 0, __ru_nswap_word = 0}, {ru_inblock = 0, __ru_inblock_word = 0}, {ru_oublock = 0, __ru_oublock_word = 0}, {ru_msgsnd = 0, __ru_msgsnd_word = 0}, {ru_msgrcv = 0, __ru_msgrcv_word = 0}, {ru_nsignals = 0, __ru_nsignals_word = 0}, {ru_nvcsw = 4, __ru_nvcsw_word = 4}, {ru_nivcsw = 1, __ru_nivcsw_word = 1}}} [07:41:18.323] record = 0x7fb3b5eebf70 [07:41:18.323] reachedRecoveryTarget = false [07:41:18.323] replayTLI = 1 [07:41:18.323] __func__ = "PerformWalRecovery" [07:41:18.323] #20 0x000056554bc66828 in StartupXLOG () at ../src/backend/access/transam/xlog.c:5723 [07:41:18.343] Insert = [07:41:18.343] checkPoint = {redo = 83886120, ThisTimeLineID = 1, PrevTimeLineID = 1, fullPageWrites = true, nextXid = {value = 749}, nextOid = 24576, nextMulti = 1, nextMultiOffset = 0, oldestXid = 730, oldestXidDB = 1, oldestMulti = 1, oldestMultiDB = 1, time = 1710661085, oldestCommitTsXid = 0, newestCommitTsXid = 0, oldestActiveXid = 749} [07:41:18.343] wasShutdown = false [07:41:18.343] didCrash = [07:41:18.343] haveTblspcMap = false [07:41:18.343] haveBackupLabel = true [07:41:18.343] EndOfLog = [07:41:18.343] EndOfLogTLI = [07:41:18.343] newTLI = [07:41:18.343] performedWalRecovery = [07:41:18.343] endOfRecoveryInfo = [07:41:18.343] abortedRecPtr = [07:41:18.343] missingContrecPtr = [07:41:18.343] oldestActiveXID = [07:41:18.343] promoted = false [07:41:18.343] __func__ = "StartupXLOG" [07:41:18.343] #21 0x000056554c1170a6 in StartupProcessMain () at ../src/backend/postmaster/startup.c:252 [07:41:18.343] No locals. [07:41:18.343] #22 0x000056554c1075bf in AuxiliaryProcessMain (auxtype=auxtype@entry=B_STARTUP) at ../src/backend/postmaster/auxprocess.c:91 [07:41:18.343] __func__ = "AuxiliaryProcessMain" [07:41:18.343] #23 0x000056554c112360 in StartChildProcess (type=type@entry=B_STARTUP) at ../src/backend/postmaster/postmaster.c:5336 [07:41:18.343] pid = [07:41:18.343] __func__ = "StartChildProcess" [07:41:18.343] #24 0x000056554c116834 in PostmasterMain (argc=argc@entry=4, argv=argv@entry=0x56554efc2e20) at ../src/backend/postmaster/postmaster.c:1470 [07:41:18.343] opt = [07:41:18.343] status = [07:41:18.343] userDoption = [07:41:18.343] listen_addr_saved = false [07:41:18.343] output_config_variable = [07:41:18.343] __func__ = "PostmasterMain" [07:41:18.343] #25 0x000056554bf41b06 in main (argc=4, argv=0x56554efc2e20) at ../src/backend/main/main.c:197 [07:41:18.343] do_check_root = [07:41:18.343] [07:41:18.343] Thread 1 (LWP 32873): [07:41:18.343] #0 0x00007fb3b8c9afc7 in sigprocmask () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.343] No symbol table info available. [07:41:18.343] #1 0x00007fb3b8d4c13a in ?? () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.343] No symbol table info available. [07:41:18.343] #2 0x00007fb3b8d5da6f in clone () from /lib/x86_64-linux-gnu/libc.so.6 [07:41:18.343] No symbol table info available.